Skip to content

Run remote-bench end-to-end via GitHub Actions and design the InferenceX-app ingest trigger #28

Description

@aistackdev

Summary

Follow-up to #26/#27 (remote-bench recipe support). Once #27 merges, the remaining gap is: remote-bench.yml produces a real, ingest-shaped agg_*.json artifact, but nothing currently pushes that artifact into InferenceX-app. Local sweeps get there via run-sweep.yml's trigger-agentic-ingest job (a repository_dispatch call with event_type: ingest-agentic-results) — remote-bench.yml is a standalone workflow_dispatch entrypoint that never runs through run-sweep.yml, so that trigger never fires for a remote-bench run today.

Scope

  1. Run the full flow for real — dispatch remote-bench.yml via GitHub Actions (not the manual SSH debug loop used for feat: support benchmarking remote/existing inference endpoints (BYO endpoint) #27's smoke test) against a live endpoint, end to end, once merged to vng-benchmark/main.
  2. Design the ingest trigger for remote-bench results. Options to weigh (not deciding yet, that's this issue):
    • Add a trigger-agentic-ingest-equivalent job directly to remote-bench.yml, firing the same repository_dispatch on success.
    • Keep remote-bench results out of the automatic ingest path entirely for now (manual/ad-hoc dispatch implies non-canonical/exploratory runs) and revisit once there's real demand for them to show up in the dashboard alongside official sweep results.
  3. Known ingest-side landmines to design around, found during feat: support benchmarking remote/existing inference endpoints (BYO endpoint) #27:
    • REMOTE_RUNNER_TYPE must resolve via InferenceX-app's hwToGpuKey() — fine for real datacenter GPUs (h200-nv etc.), but the dev endpoint we smoke-tested against (RTX 5090) isn't in GPU_KEYS at all, so that specific result wouldn't resolve a GPU key on ingest. Not a blocker for real targets, but means we can't fully validate the ingest path against that same dev box.
    • image must be the real deployed image (fixed in feat: support benchmarking remote/existing inference endpoints (BYO endpoint) #27) — otherwise ingest records a meaningless placeholder.
    • Duration matters for canonical-eligibility: --unsafe-override kicks in under 900s (as our smoke tests all were), which the aggregator likely flags as non-canonical the same way it does for local recipes — confirm whether InferenceX-app's ingest treats that the same way for remote-bench rows.

Not in scope

No InferenceX-app schema/mapper changes assumed yet — this issue is scoped to deciding whether/how the trigger should exist, not implementing it. If a mapper change turns out to be needed, that becomes a follow-up PR against SemiAnalysisAI/InferenceX-app, coordinated separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions