You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #26/#27 (remote-bench recipe support). Once #27 merges, the remaining gap is: remote-bench.yml produces a real, ingest-shaped agg_*.json artifact, but nothing currently pushes that artifact into InferenceX-app. Local sweeps get there via run-sweep.yml's trigger-agentic-ingest job (a repository_dispatch call with event_type: ingest-agentic-results) — remote-bench.yml is a standalone workflow_dispatch entrypoint that never runs through run-sweep.yml, so that trigger never fires for a remote-bench run today.
Design the ingest trigger for remote-bench results. Options to weigh (not deciding yet, that's this issue):
Add a trigger-agentic-ingest-equivalent job directly to remote-bench.yml, firing the same repository_dispatch on success.
Keep remote-bench results out of the automatic ingest path entirely for now (manual/ad-hoc dispatch implies non-canonical/exploratory runs) and revisit once there's real demand for them to show up in the dashboard alongside official sweep results.
REMOTE_RUNNER_TYPE must resolve via InferenceX-app's hwToGpuKey() — fine for real datacenter GPUs (h200-nv etc.), but the dev endpoint we smoke-tested against (RTX 5090) isn't in GPU_KEYS at all, so that specific result wouldn't resolve a GPU key on ingest. Not a blocker for real targets, but means we can't fully validate the ingest path against that same dev box.
Duration matters for canonical-eligibility: --unsafe-override kicks in under 900s (as our smoke tests all were), which the aggregator likely flags as non-canonical the same way it does for local recipes — confirm whether InferenceX-app's ingest treats that the same way for remote-bench rows.
Not in scope
No InferenceX-app schema/mapper changes assumed yet — this issue is scoped to deciding whether/how the trigger should exist, not implementing it. If a mapper change turns out to be needed, that becomes a follow-up PR against SemiAnalysisAI/InferenceX-app, coordinated separately.
Summary
Follow-up to #26/#27 (remote-bench recipe support). Once #27 merges, the remaining gap is:
remote-bench.ymlproduces a real, ingest-shapedagg_*.jsonartifact, but nothing currently pushes that artifact into InferenceX-app. Local sweeps get there viarun-sweep.yml'strigger-agentic-ingestjob (arepository_dispatchcall withevent_type: ingest-agentic-results) —remote-bench.ymlis a standaloneworkflow_dispatchentrypoint that never runs throughrun-sweep.yml, so that trigger never fires for a remote-bench run today.Scope
remote-bench.ymlvia GitHub Actions (not the manual SSH debug loop used for feat: support benchmarking remote/existing inference endpoints (BYO endpoint) #27's smoke test) against a live endpoint, end to end, once merged tovng-benchmark/main.trigger-agentic-ingest-equivalent job directly toremote-bench.yml, firing the samerepository_dispatchon success.REMOTE_RUNNER_TYPEmust resolve via InferenceX-app'shwToGpuKey()— fine for real datacenter GPUs (h200-nvetc.), but the dev endpoint we smoke-tested against (RTX 5090) isn't inGPU_KEYSat all, so that specific result wouldn't resolve a GPU key on ingest. Not a blocker for real targets, but means we can't fully validate the ingest path against that same dev box.imagemust be the real deployed image (fixed in feat: support benchmarking remote/existing inference endpoints (BYO endpoint) #27) — otherwise ingest records a meaningless placeholder.--unsafe-overridekicks in under 900s (as our smoke tests all were), which the aggregator likely flags as non-canonical the same way it does for local recipes — confirm whether InferenceX-app's ingest treats that the same way for remote-bench rows.Not in scope
No InferenceX-app schema/mapper changes assumed yet — this issue is scoped to deciding whether/how the trigger should exist, not implementing it. If a mapper change turns out to be needed, that becomes a follow-up PR against
SemiAnalysisAI/InferenceX-app, coordinated separately.