Migrate agentic-coding benchmarks to aiperf v0.2 (reopened)

[cquil11]

Summary

Migrates scenario-type: agentic-coding from kv-cache-tester to aiperf (cjq/weka-live-assistant-responses), adds Kimi K2.5 agentic sweeps across B200, B300, H100, H200, MI355X, and pulls in a stack of agentic-specific aiperf features. bmk_agentic JSON schema preserved — downstream aggregators unaffected.

Dataset (v0.2): semianalysisai/cc-traces-weka-no-subagents-051226. Pulled via aiperf's --public-dataset semianalysis_cc_traces_weka flag — the loader name is a stable alias for the dated HF revision, so bumping the dataset means just rev'ing the alias on the aiperf side.

InferenceX repo additions since v0.1

• benchmarks/benchmark_lib.sh: new build_replay_cmd emits aiperf profile --scenario inferencex-agentx-mvp … with --streaming --use-server-token-count --random-seed 42 --failed-request-threshold 0.05 --trajectory-start-{min,max}-ratio 0.25/0.75; install_agentic_deps editable-installs aiperf in-container; AIPERF_DATASET_{CONFIGURATION_TIMEOUT,MMAP_CACHE_DIR,WEKA_LIVE_ASSISTANT_RESPONSES} wired
• utils/process_agentic_result.py rewritten to consume aiperf profile_export.jsonl + profile_export_aiperf.json + server_metrics_export.json; preserves all keys summarize.py consumes; theoretical cache-hit computed from trace metadata
• utils/summarize.py per-metric stat set: drop median / p99 / p99.9, add p75 / p95 (keep mean / p90 / std)
• utils/test_process_agentic_result.py fixture-driven contract test for every key downstream reads
• 6 new agentic launchers: kimik2.5_{fp4_b200, fp4_b300, fp4_mi355x, int4_b200, int4_h100, int4_h200}.sh with per-target tunings (fp8 KV on Hopper, --block-size=1 + AITER on MI355X, lazy_offload=true JSON form on H200+MI355X, VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 on B200 for TP=4)
• .github/configs/{nvidia,amd}-master.yaml: agentic-coding blocks for all 5 SKUs with explicit conc-list per offloading mode + runner: h200-dgxc pin (cache mount availability)
• runners/launch_h200-dgxc-slurm.sh: mount /home/sa-shared/gharunners/ai-perf-cache → /aiperf_mmap_cache, export AIPERF_DATASET_MMAP_CACHE_DIR (parity with B200 DGXC + MI355X)
• Removed: utils/find_reusable_sweep_run.py, utils/validate_reusable_sweep_artifacts.py + tests (aiperf's content-addressed mmap cache replaces them)

aiperf submodule additions since v0.1

• inferencex-agentx-mvp scenario plugin: locks --num-dataset-entries, --inter-turn-delay-cap-seconds 60, --cache-bust first_turn_prefix, timing_mode=AGENTIC_REPLAY, requires --ignore-eos and weka_trace loader
• AGENTIC_REPLAY timing strategy + TrajectorySource: per-trajectory k_i sampling, WARMUP→PROFILING resume at k_i+1, FIFO recycle queue, context-overflow trajectory short-circuit
• weka_trace loader with AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES=1 mode (user-only deltas, server-threaded assistant responses) — sources from semianalysisai/cc-traces-weka-no-subagents-051226
• ContextOverflowCountMetric + scenario.context_overflow.is_context_overflow_response substring classifier; RecordProcessor drops overflow records before metrics push in AGENTIC_REPLAY scenarios
• --failed-request-threshold (float in [0,1]): in-flight ProfileCancelCommand broadcast when error rate exceeds threshold after max(concurrency, 10) grace floor
• --trajectory-start-{min,max}-ratio: replaces hardcoded 0%-70% k_i range; seed-deterministic per (random_seed, trace_id)
• TrajectorySource build-time summary table (range cfg + per-lane k_i/N/pct/trace_id); per-trajectory warmup-completion log lines on credit return
• Content-addressed mmap dataset cache (AIPERF_DATASET_MMAP_CACHE_DIR); AIPERF_DATASET_CONFIGURATION_TIMEOUT + AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT env knobs (default 300s → 1800s in our launchers)
• --use-server-token-count: derive ISL/OSL from server usage.prompt_tokens/completion_tokens instead of client tokenizer (avoids CPU-pinning at high conc on custom-tokenizer models)
• Realtime stats row: srv line with server-side running-avg throughput; intvty per-user row replaces split tin/tout; ISL/OSL p50/p75/p90/p99 row; per-warmup-completion lines emitted in non-TTY runs

Test plan

• Kimi K2.5 INT4 H200 (lazy_offload + cache mount): 17/17 clean
• Kimi K2.5 FP4 B200 TP=8 + TP=4 (cudagraph estimator off): 24/24 clean
• Kimi K2.5 FP4 B300 none: 9/9 clean
• Kimi K2.5 FP4 B300 cpu: blocked on cluster recovery
• Kimi K2.5 FP4 MI355X: re-dispatch after g09 + g11 drained

Results

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://docs.sglang.io/cookbook/intro
• https://recipes.vllm.ai/

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sglang/tree/main/docs_new

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

[claude]

🔴 The new write_agentic_result_json calls python3 $INFMAX_CONTAINER_WORKSPACE/utils/generate_aiperf_plots.py "$result_dir" to produce results/metrics_plots.png, and the workflow lists that PNG in the agentic artifact upload bundle — but utils/generate_aiperf_plots.py is not present anywhere in this PR or the repo. The call is wrapped in 2>&1 || true and the upload uses if-no-files-found: ignore, so the launcher and workflow appear green while every agentic run silently fails to emit the advertised plot. Either commit the missing script or remove the invocation at benchmarks/benchmark_lib.sh:1037 and the results/metrics_plots.png line at .github/workflows/benchmark-tmpl.yml:248.

Extended reasoning...

What the bug is

write_agentic_result_json in benchmarks/benchmark_lib.sh (lines 1026-1038) runs the post-run aggregator and then invokes a plotter:

python3 "$INFMAX_CONTAINER_WORKSPACE/utils/generate_aiperf_plots.py" "$result_dir" 2>&1 || true

This function is the final step of every agentic launcher under benchmarks/single_node/agentic/*.sh in this PR. Its output results/metrics_plots.png is then listed as one of the files in the agentic artifact upload at .github/workflows/benchmark-tmpl.yml:248. The comment block immediately above the plotter call advertises it as an intentional feature (Generate metrics_plots.png from the same aiperf artifacts), and the build_replay_cmd comment at lines 1008-1014 explicitly justifies the 1-second --slice-duration "so the post-run plotter has per-window time series … Without this, aiperf only emits aggregate stats and the 6x2 panels collapse to flat lines."

But utils/generate_aiperf_plots.py is not committed anywhere in the PR or the repo. A repo-wide search finds exactly one reference — the invocation itself — and no file matching **/generate_aiperf_plots*.

Why nothing currently fails

The call is wrapped in 2>&1 || true, so python: can't open file '.../utils/generate_aiperf_plots.py': [Errno 2] No such file or directory is captured into benchmark.log and the launcher's exit status stays clean. Separately, the workflow upload step uses if-no-files-found: ignore, so the missing results/metrics_plots.png is silently dropped from the uploaded bundle. There is no other check that would surface the missing artifact.

Step-by-step proof

1. An agentic-coding job runs, e.g. benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh.
2. The script ends with write_agentic_result_json "$RESULT_DIR" (line 156 of the new launcher).
3. write_agentic_result_json invokes python3 $INFMAX_CONTAINER_WORKSPACE/utils/generate_aiperf_plots.py "$result_dir".
4. Python exits non-zero with No such file or directory because the script does not exist.
5. || true swallows the exit code; only benchmark.log records the error.
6. The aggregated *.json is created, so the workflow's retry-based existence check at .github/workflows/benchmark-tmpl.yml passes.
7. The Upload agentic raw results step lists results/metrics_plots.png but the file does not exist; if-no-files-found: ignore causes the missing file to be silently skipped.
8. The artifact bundle is published without the advertised PNG, and no warning surfaces in the run summary.

Impact

The metrics_plots.png artifact is advertised in the PR description (build_replay_cmd comment) and explicitly listed in the workflow upload, but it will never be produced for any agentic run. Every agentic benchmark.log will also carry a noisy python: can't open file … line, complicating future log triage. This is not a correctness bug for the agg JSON path (the success gate is the JSON, not the PNG), so the pipeline still goes green — but the promised per-window time-series visualization is missing from every run.

How to fix

Two options, either is sufficient:

1. Commit utils/generate_aiperf_plots.py alongside this PR. The 1-second --slice-duration plumbing in build_replay_cmd and the workflow artifact reference were clearly added in anticipation of this file.
2. Remove the plotter call from write_agentic_result_json (benchmarks/benchmark_lib.sh:1034-1037) and drop results/metrics_plots.png from .github/workflows/benchmark-tmpl.yml:248. The --slice-duration 1.0 flag in build_replay_cmd can also be removed if no other consumer needs the per-window timeslice JSON, but profile_export_aiperf_timeslices.{json,csv} are also in the upload bundle, so it may still be useful.

Option 1 matches the apparent intent (the launcher comment is written assuming the plotter exists). Option 2 is the safer "make the PR self-consistent" path.

[claude]

🔴 utils/process_agentic_result.py:40 hardcodes _HF_DATASET = "semianalysisai/cc-traces-weka-042026" (the v0.1 name) while benchmarks/benchmark_lib.sh:908 in this same PR downloads semianalysisai/cc-traces-weka-no-subagents-051226 (v0.2, matching the PR description and the aiperf submodule loader). _hf_traces_dir() builds its lookup path from _HF_DATASET, so the production HF cache directory datasets--semianalysisai--cc-traces-weka-no-subagents-051226/snapshots/ is never found, _load_trace_metadata() returns {}, and every shipped agentic agg JSON has theoretical_cache_hit_rate=null with all output_tokens_expected stats missing — silently breaking the "theoretical cache-hit computed from trace metadata" feature this PR advertises. Fix is a one-line constant bump (and the matching test fixture path at utils/test_process_agentic_result.py:408); ideally derive the name from a shared constant / env var shared with resolve_trace_source.

Extended reasoning...

What goes wrong

Two locations in this PR disagree on the HF dataset name:

• Producer — benchmarks/benchmark_lib.sh:908 (resolve_trace_source) calls hf download --repo-type dataset semianalysisai/cc-traces-weka-no-subagents-051226. This is the only producer of the local HF cache for the agentic path, and it matches the PR description ("v0.2: semianalysisai/cc-traces-weka-no-subagents-051226") and the aiperf submodule loader (semianalysis_cc_traces_weka is the stable alias for this dated revision).
• Consumer — utils/process_agentic_result.py:40 hardcodes _HF_DATASET = "semianalysisai/cc-traces-weka-042026" — the v0.1 name from before this PR.

_hf_traces_dir() (process_agentic_result.py:118–146) builds its lookup path from _HF_DATASET:

org, name = _HF_DATASET.split("/", 1)
snapshots = cache_root / f"datasets--{org}--{name}" / "snapshots"

So it searches $HF_HUB_CACHE/datasets--semianalysisai--cc-traces-weka-042026/snapshots/, which never exists in production. The function returns None, _load_trace_metadata() returns an empty dict, and:

1. compute_cache_stats() (process_agentic_result.py ~395–432): the theoretical-cache-hit walk runs only if metadata is truthy. Empty dict ⇒ result["theoretical_cache_hit_rate"] stays None.
2. compute_workload_stats() (process_agentic_result.py ~277–294): the mean/p75/p90/p95/std_output_tokens_expected block is gated by if metadata:. Empty dict ⇒ none of those keys are emitted.

Impact

Every agentic-coding result JSON shipped to the downstream aggregator has theoretical_cache_hit_rate: null and is missing all five output_tokens_expected stats — directly contradicting this PR's claim: "theoretical cache-hit computed from trace metadata." The per-launcher print(f" Theoretical cache hit rate: ...") at the end of process_agentic_result.py simply never prints in production. End-users consuming the schema downstream see silent data loss, not a crash.

Why CI doesn't catch this

The new unit test utils/test_process_agentic_result.py:408 (test_processor_loads_traces_jsonl_for_theoretical_cache) builds its fake HF cache snapshot under hf_cache / "datasets--semianalysisai--cc-traces-weka-042026" / "snapshots" / "abc" — mirroring the same stale name the production code reads. The test passes because both sides agree on the wrong name. Nothing else exercises _HF_DATASET against the real dataset, and test_processor_emits_required_summarize_keys (line 264) doesn't include the optional theoretical_cache_hit_rate / output_tokens_expected keys in its required-key set, so their absence isn't flagged either.

Step-by-step proof

1. A B200 job runs benchmarks/single_node/agentic/kimik2.5_fp4_b200.sh, which calls resolve_trace_source (benchmark_lib.sh:907–916).
2. That downloads semianalysisai/cc-traces-weka-no-subagents-051226 into $HF_HUB_CACHE. After download, $HF_HUB_CACHE contains datasets--semianalysisai--cc-traces-weka-no-subagents-051226/snapshots/<rev>/traces.jsonl (the HF hub layout).
3. The launcher runs aiperf, which writes results/trace_replay/profile_export.jsonl etc.
4. write_agentic_result_json invokes python3 utils/process_agentic_result.py.
5. _load_trace_metadata() calls _hf_traces_dir(), which computes snapshots = $HF_HUB_CACHE/datasets--semianalysisai--cc-traces-weka-042026/snapshots from _HF_DATASET. snapshots.is_dir() is False (the cache directory uses the real dataset name -no-subagents-051226). Returns None.
6. _load_trace_metadata() returns {}. _TRACE_METADATA_CACHE = {}.
7. In compute_cache_stats(), if metadata: is False ⇒ theoretical_cache_hit_rate stays None.
8. In compute_workload_stats(), the trace-metadata block at the bottom of the function never executes ⇒ no *_output_tokens_expected keys are written.
9. Aggregator emits agg_*.json with "theoretical_cache_hit_rate": null and no expected-output stats. The per-launcher print(" Theoretical cache hit rate: ...") is gated on agg.get("theoretical_cache_hit_rate") is not None, so it never prints.

How to fix

Cheapest fix is a one-line constant bump in utils/process_agentic_result.py:

_HF_DATASET = "semianalysisai/cc-traces-weka-no-subagents-051226"

Plus the matching path in utils/test_process_agentic_result.py:408 (hf_cache / "datasets--semianalysisai--cc-traces-weka-no-subagents-051226" / "snapshots" / "abc"). Better: read the dataset name from a shared module-level constant (or env var) that both resolve_trace_source and process_agentic_result.py consume, so future dataset bumps can't desync again. As a regression guard, add an assertion in the test that the constant equals what resolve_trace_source actually downloads — or simply have the test set _HF_DATASET via the env-based shared constant rather than hardcoding the path.

[claude]

🟡 build_agg() in utils/process_agentic_result.py sets both "num_requests_total" and "num_requests_successful" to len(records), but load_records() above explicitly filters out error records via 'if obj.get("error"): continue'. The two fields are therefore always equal — any downstream consumer computing a failure rate as 1 - successful/total will see 0% even when aiperf actually had errored requests. Fix is small: count error records during load_records and surface the count, or pull the true total from the aiperf aggregate JSON (request_count is already loaded but unused for these keys).

Extended reasoning...

What the bug is

utils/process_agentic_result.py rewrites the legacy agg JSON for the aiperf-based agentic pipeline. The legacy CSV-based implementation correctly distinguished the two fields:

# legacy (removed in this PR)
"num_requests_total": len(rows),         # all CSV rows
"num_requests_successful": len(successful),  # filtered by success == 'True'

The new code sets both to the same value:

# utils/process_agentic_result.py (build_agg, ~lines 601-603)
"num_requests_total": len(records),
"num_requests_successful": len(records),

But records is the output of load_records() (lines 93–105), which explicitly drops error rows:

obj = json.loads(line)
if obj.get("error"):
    continue
records.append(obj)

So records only contains successful requests; the two emitted fields are mathematically identical for every run.

Step-by-step proof

1. aiperf writes profile_export.jsonl with one line per request. Failed requests carry an error key (verified in aiperf source).
2. load_records() reads that file and skips any line where obj.get("error") is truthy.
3. build_agg() then does "num_requests_total": len(records) and "num_requests_successful": len(records) — same records list.
4. Imagine a run with 950 records emitted by aiperf, 50 of which carry "error": "...". load_records() returns a list of 900 entries. The agg JSON now claims num_requests_total = 900 and num_requests_successful = 900, so any consumer computing 1 - successful/total reads 0% failure rate even though aiperf saw 50 failures.

Why existing code doesn't prevent it

aiperf's --failed-request-threshold 0.05 aborts the run upstream above 5% error rate, but that's a coarse gate — non-zero error counts below 5% still pass through silently, and the labels still lie about the true total. The aggregate JSON (profile_export_aiperf.json) IS loaded by this script and contains request_count (the true total including errors) but it is never consulted for these two keys.

Impact

In-repo blast radius is bounded: utils/summarize.py headers don't read these fields, and the only consumer asserting on them is utils/test_process_agentic_result.py:491. The two fields nominally exist to track distinct quantities — that's why the pre-PR code emitted them separately — so any external dashboard / script (downstream Pareto analysis, the team's spreadsheets, etc.) computing failure rate from the emitted schema will see 0% regardless of actual aiperf error count. This is a silent schema regression.

How to fix

Two clean options:

1. Count errors during loading. Change load_records to return (records, error_count) (or a second pass), then in build_agg:

   "num_requests_total": len(records) + error_count,
   "num_requests_successful": len(records),

2. Pull from the aggregate. aggregate["request_count"] is already loaded:

   "num_requests_total": aggregate.get("request_count", len(records)),
   "num_requests_successful": len(records),

Either restores the original semantics of the two-field design.

[claude]

🟡 compute_cache_stats._final_value() sums values across all per-engine metric series, which is correct for counters (vllm:prefix_cache_hits/queries, vllm:prompt_tokens, vllm:kv_offload_bytes_*) but wrong for the one percentage gauge in the mapping at lines 489–498 — vllm:cpu_kv_cache_usage_perc → cpu_kv_cache_usage_pct. With the DP-attn configs this PR adds (dsv4-fp4-b200-vllm dp-attn:true, dsv4-fp4-b300-vllm, dsv4-fp8-h200-vllm), vLLM emits one /metrics series per DP engine, so two engines at 50% report 100% and eight at 50% report 400%. Easiest fix: pick avg (not total) for percentage gauges, e.g. a small per-metric strategy table keyed on metric name.

Extended reasoning...

What the bug is. _final_value() walks ("total", "max", "avg") and, for the first stats key any series has, accumulates agg += float(v) across every series entry. The intent (see the comment at lines 447–448: "We aggregate across series (multiple endpoints / label sets) and prefer total for counters, then max/avg for gauges") is to sum counters and aggregate gauges, but the implementation sums in both cases.

Which metric is affected. Of the five metrics looked up via this helper for compute_cache_stats:

• vllm:prefix_cache_hits / vllm:prefix_cache_queries — counters. Summing across series (and dividing in lockstep at lines 481–482) is correct.
• vllm:cpu_prefix_cache_hits / vllm:cpu_prefix_cache_queries — counters, same lockstep ratio.
• vllm:prompt_tokens / vllm:generation_tokens — counters.
• vllm:kv_offload_bytes_* / vllm:kv_offload_time_* — counters.
• vllm:cpu_kv_cache_usage_perc → cpu_kv_cache_usage_pct — a gauge expressing a percentage. This one is wrong when there are multiple series.

Why DP-attn configs trigger it. In a single-engine layout (TP-only), vLLM exposes one /metrics series, so the loop sums a single value and the result is right by accident. With DP-attn (added in this PR for dsv4-fp4-b200-vllm, dsv4-fp4-b300-vllm, dsv4-fp8-h200-vllm), each DP engine runs its own scheduler and emits its own series — typically tagged by engine index. The loop now sums N independent gauge readings.

Step-by-step proof. Take the dsv4-fp8-h200-vllm agentic-coding entry added in this PR:

- { tp: 8, ep: 8, dp-attn: true, offloading: none, conc-list: [1, 2, 4, 8, 16] }

The launcher (dsv4_fp8_h200.sh) runs --data-parallel-size 8 → 8 DP engines, each exporting vllm:cpu_kv_cache_usage_perc. Say each engine has its CPU offload pool 50% full. The scrape produces a metrics dict shaped like:

"vllm:cpu_kv_cache_usage_perc": {
  "series": [
    {"labels": {"engine": "0"}, "stats": {"max": 50.0, "avg": 50.0, ...}},
    {"labels": {"engine": "1"}, "stats": {"max": 50.0, "avg": 50.0, ...}},
    ... 6 more identical entries ...
  ]
}

_final_value() tries "total" first — not present on a gauge — then "max", which is present on every series. It iterates and accumulates agg = 50 + 50 + 50 + 50 + 50 + 50 + 50 + 50 = 400.0, then returns 400.0. That value is written to cpu_kv_cache_usage_pct in the agg_*.json payload. The true aggregate utilization is ~50%, not 400%.

Impact. cpu_kv_cache_usage_pct is a diagnostic field for analyzing CPU-offload behaviour, not a headline metric. utils/summarize.py doesn't read it (verified — it consumes only the throughput/latency keys plus topology fields), so summary tables are unaffected. But the field is published in the per-run JSON and would mislead anyone inspecting the offload regime of the new DP-attn configs — exactly the configs this PR is intended to characterize. The misreport is also obviously wrong (>100%, often by a large multiple) so it's likely to be spotted, but it crowds noise into the very signal the new sweeps are designed to produce.

Suggested fix. Cheapest correct version is a per-metric aggregation strategy table, e.g.:

_AGG = {
    "vllm:cpu_kv_cache_usage_perc": "avg",
    # everything else defaults to summing "total" -> "max" -> "avg"
}

def _final_value(metric_name):
    mode = _AGG.get(metric_name, "sum")
    ...
    if mode == "avg":
        # mean across series of stats["avg"] (or "max" if "avg" missing)
    else:
        # existing sum behavior

Alternatively, pick avg instead of max when the only key available is a gauge stat — but the table form is clearer about which metrics are gauges. Existing tests (test_processor_aggregates_across_multiple_series) cover only the counter case; adding a percentage-gauge fixture would lock the behaviour in.