Skip to content

fix(ci): a PR based on any branch but main runs no unit tests, no lint and no agent suites — and looks green #2767

Description

@itomek

A pull request opened against anything other than main runs almost no CI — no unit
tests, no lint, no agent suites — and its checks page looks healthy the whole time. Nothing is
marked failed or skipped-with-a-warning; the jobs simply never exist.

This is not theoretical. Measured on merged PRs:

PR base total checks of which email / unit / lint
#2599 (email work) tmi/2580-attention-int 16 0
#2660 tmi/issue-2621-grounded-prose 11 0
#2662 tmi/issue-2643-2638-scan-cost 11 0
#2757 (control) main 48 8

#2599 changed email-agent code and merged with zero email tests, zero unit tests and zero
lint.
The stacked PRs kept the handful of workflows that don't filter on base (CodeQL and
friends), which is exactly what makes this dangerous — a green checks page that means nothing.

Cause: on.pull_request.branches filters on the PR's base branch, not the head. So
branches: [ main ] means "only when merging into main". Every stacked PR falls outside it.

.github/workflows/test_email_agent.yml:30-31:

  pull_request:
    branches: [ main ]

35 workflows carry the same filter, including lint.yml, test_unit.yml,
test_security.yml and dependency-review.yml — so the gap is the entire test matrix, not one
agent's suite.

This is the third way a PR can silently skip its checks in this repo, alongside #2755
(ready_for_ci cannot re-trigger skipped jobs, because labeled is not a PR trigger type).
The pattern is the problem: every one of them fails open and quiet. Stacking PRs is a
normal workflow here — seven merged PRs used a non-main base — so this is not an edge case.

🔍 Fix options, and the one to avoid

Don't just delete the branches: filter. These workflows also trigger on push to main,
and dropping the PR-side filter without thinking through the pairing gets you duplicate runs on
every branch push and a heavier runner bill.

Two directions worth weighing:

  1. Widen the filter — e.g. branches-ignore: [], or an explicit allowlist that includes
    the feature-branch naming conventions actually in use (tmi/*, issue-*, claudia/*).
    Smallest diff, but it needs applying consistently across all 35 files or the gap just moves.
  2. A required aggregating gate job that runs unconditionally and fails when its expected
    dependencies did not run. This is the one that closes the whole class — it converts every
    silent skip, including fix(ci): ready_for_ci label does not re-trigger skipped jobs — 'labeled' is not a PR trigger type #2755's and any future variant, into a visible red check.

Option 2 is the durable answer; option 1 is worth doing first if a quick stop-gap is wanted.

Verification: open a throwaway PR against a non-main base that touches
hub/agents/email/python/**, and confirm the email suites appear in its checks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingp0high priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions