Skip to content

Finish submission checker log parsing in both endpoint logs and loadgen logs #2636

Merged
pgmpablo157321 merged 1 commit into
mlcommons:update_endpoints_structurefrom
zihaok:zihaok-pr2630-submission-checker-fix
Jul 20, 2026
Merged

Finish submission checker log parsing in both endpoint logs and loadgen logs #2636
pgmpablo157321 merged 1 commit into
mlcommons:update_endpoints_structurefrom
zihaok:zihaok-pr2630-submission-checker-fix

Conversation

@zihaok

@zihaok zihaok commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

MR Enhancement Summary

  • Enhanced endpoint submission parsing to read latency, TTFT, and TPOT from early_stopping_percentiles instead of the normal percentile fields. please see this MR in endpoint repo feat(metrics): opt-in MLPerf early-stopping percentile estimates (LoadGen parity) endpoints#417 that implement early stopping in endpoint
  • Updated endpoint LLM latency validation so LLM models collected both in endpoint and loadgen use TTFT/TPOT limits, while non-LLM models collected using endpoint or loadgen continue to use end-to-end latency checks.
  • Bugfix: Added gpt-oss-120b to the LLM model set so it follows the same TTFT/TPOT validation path as DeepSeek-R1.
  • Fixed endpoint accuracy sample-count handling so accuracy coverage is derived from accuracy/accuracy_results.json, not from performance n_samples_issued or n_samples_completed.
  • Added scenario-specific Qwen3-VL accuracy sample-count handling to avoid applying one dataset-size requirement across all endpoint scenarios. Server: 48k, Interactive: 8K
  • Tightened endpoint compliance-test handling to only include test04 in Wan, instead of checking every benchmark's audit test folder. gptoss and dsr1 doesn't need separate audit test.
  • Improved endpoint parser robustness for nested JSON paths and percentile keys such as 99.0 and 99.9.
  • in Endpoint log parsing, enforce accuracy dataset has a sample count that is the same as dataset whole size

@zihaok
zihaok requested review from a team as code owners July 18, 2026 06:35
@github-actions

Copy link
Copy Markdown
Contributor

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@zihaok zihaok changed the title Fix endpoints submission checker latency parsing Finish submission checker log parsing in both endpoint logs and loadgen logs Jul 18, 2026
@pgmpablo157321

Copy link
Copy Markdown
Contributor

I have tested it with the sample logs and looks good

@pgmpablo157321
pgmpablo157321 merged commit a055a57 into mlcommons:update_endpoints_structure Jul 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants