Skip to content

Nothing holds main to the gates a branch must pass, and the red lands on the next unrelated branch #2950

Description

@localai-org-maint-bot

Row: GATE-CI-CONCURRENCY

What

scripts/agent-preflight.sh runs a set of cheap record gates BEFORE a push, from
a branch. Nothing holds main to those same gates. So a change that breaks one
lands green, and the red is then charged to whichever unrelated branch runs
preflight next.

ci.yml's own header already states the structural half (lines 17-33): every
expensive job carries a job-level group keyed on github.ref, which is the
constant refs/heads/main for every push, so consecutive pushes cancel each
other. Measured there over 40 consecutive main runs at 0eb049f7: 26 cancelled,
12 failure, 1 success.

The instances

  1. check-env-doc is RED on main: #2304 landed VT_DFLASH_BOUNDS_DEVICE undocumented #2312VT_DFLASH_BOUNDS_DEVICE landed undocumented; check-env-doc red on
    an unrelated branch.
  2. check-env-doc is RED on main again: #2349 landed VT_QWEN35_STAGE_RESERVE_BYTES undocumented #2356VT_QWEN35_STAGE_RESERVE_BYTES, same gate, same shape, one commit
    later. This one landed with no pull request at all.
  3. windows-msvc is RED on every PR: glm5_next_weights.cpp shadows 'v' and /WX makes it an error #2404glm5_next_weights.cpp shadowed v; windows-msvc is PR-only with
    no main baseline, so it landed green and reddened every subsequent PR.
  4. test_check_gate_commands is RED on main: ENG-PREFLIGHT-COMPILES became runnable without re-pinning RUNNABLE_BASELINE #2429ENG-PREFLIGHT-COMPILES became runnable without re-pinning
    RUNNABLE_BASELINE; test_check_gate_commands red on an unrelated branch.
  5. The api_server STATUS_STACK_BUFFER_OVERRUN crash, hidden behind windows-msvc is RED on every PR: glm5_next_weights.cpp shadows 'v' and /WX makes it an error #2404's
    compile break until it was fixed.
  6. Live, 2026-09-05: tests/tools/test_oracle_pin.py's
    test_metadata_and_runtime_strings_differ_on_the_pin is red on main at
    471dbeefd, proven by running it on an unmodified base. Tracked separately as
    main is red: an ORACLE-VLLM sync made the distribution and oracle versions equal, which test_oracle_pin pins as impossible #2931. It was charged first to an unrelated placement-gate branch.

What is written down

.agents/specs/gate-on-main.md — an analysis, not a row. It answers what gates
main today job by job, what the baseline lane's three mechanisms actually
protect, which preflight gates are cheap and deterministic enough to qualify, what
a per-push record job would cost (≈ 4 min/run, ≈ 3.1 job-hours/day), and the case
AGAINST the change.

It deliberately opens no new row. The diagnosis is already carried by
main-verifiability.md and ci.yml's header; what is new is one design option.

The recommendation, staged cheapest first

  1. Use workflow_dispatch for two weeks. It costs nothing and has been used once
    ever.
  2. If that does not close the gap, add record-gates-main: if: push, group
    keyed on github.sha, cancel-in-progress: false, sharing one script with
    agent-record.
  3. Revisit required status checks once there is completion data.

Measured to catch 3 of the 5 historical instances at 1.6% of the cost #274
rejected. The other two are the Windows lane and need a different answer.

Caveat carried from the verification pass

Four numbers are marked unverified in the document (a suite total and three local
checker timings) because the box was at load ~120 and a re-timing would not have
been comparable. 26 of the 31 issue numbers the analysis cites, including #274
and #584, return 404 to an unauthenticated reader
, so several of its citations
cannot be followed by anyone outside the repo.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions