Skip to content

fix(trigger): recognize Google ADK tools=[...] as a tool-surface change - #349

Merged
pengfei-threemoonslab merged 2 commits into
mainfrom
claude/github-issue-315-df057d
Aug 8, 2026
Merged

fix(trigger): recognize Google ADK tools=[...] as a tool-surface change#349
pengfei-threemoonslab merged 2 commits into
mainfrom
claude/github-issue-315-df057d

Conversation

@pengfei-threemoonslab

@pengfei-threemoonslab pengfei-threemoonslab commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fixes bug(trigger): recognize Google ADK LlmAgent tools=[...] changes as tool-surface changes #315. A Google ADK PR whose root agent is LlmAgent(name="support_agent", tools=[lookup_account, delete_account]) — two directly reachable tools, one of them destructive — routed as skip_reason: "dry_run_only". The only rule that fired was TRIGGER-FRAMEWORK-VERSION-BUMP, on a raw google-adk string somewhere in the diff, and it reported the result as a framework upgrade.
  • Adds TRIGGER-GOOGLE-ADK-AGENT-TOOLS-CHANGED (surface_class: capability, run_shipgate). This was catalog drift, not a missing capability: detection (GOOGLE_ADK_AGENT_CLASSES = {"Agent", "LlmAgent"} in cli/discovery/signals.py), the ADK adapter (inputs/google_adk.py), and the binding graph all resolve this shape already — docs/triggers.json was the one component carrying no ADK rule at all. Plain functions handed to tools=[...] carry no decorator, so @function_tool / FunctionTool( never sees them, and ADK's most common agent spelling had no positive route. The new rule conjoins a google.adk module path, an Agent( / LlmAgent( construction, and a tools=[...] argument, using the engine's own signal vocabulary.
  • Tightens TRIGGER-FRAMEWORK-VERSION-BUMP. Nothing about the string google-adk establishes that a dependency version moved; it comes just as easily from install prose or a sample import, which is why a docs-only change could be classified as a framework upgrade. The rule now requires both halves of its claim — the package token and a changed dependency manifest (pyproject.toml, requirements*.txt, package.json, a lockfile, pom.xml, …) — and its rationale reports the co-occurrence it observed rather than the upgrade it inferred. Coverage of real bumps is unchanged.

Both changes stay inside the current catalog schema_version (0.3, as set by #340), add no predicate vocabulary, and keep every existing rule ID, so an external agent that pre-fetched triggers.json keeps working. No new public surface — this closes a gap between the catalog and the engine's existing ADK detection.

Rebased onto main after #330, #331, and #340 landed; the full suite and ruff check . were re-run green on the new base, and the reproduction below was re-confirmed there.

Type

  • Check or risk-model change
  • Input adapter change
  • CLI or GitHub Action behavior (trigger verdict and verify --preview routing)
  • Report, schema, or SARIF output
  • Documentation only

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

Additional local checks run:

  • Reproduced the reported defect and its fix against the real PR. Pulled all 9 files and their actual patches from google/adk-python#6605 via gh api and ran the evaluator against the pre-change catalog and this branch's:

    before (main) after
    should_run / run_shipgate false true
    skip_reason dry_run_only null
    carrying rule TRIGGER-FRAMEWORK-VERSION-BUMP (dry_run) TRIGGER-GOOGLE-ADK-AGENT-TOOLS-CHANGED (run)
    diff_tokens ["google-adk"] ["Agent(", "LlmAgent(", "google-adk", "google.adk", "tools=["]
  • End-to-end through the issue's own command. Built a git repo carrying the contributing/samples/agent_hooks/agent.py sample and ran agents-shipgate verify --preview --workspace . --base … --head … --format json plus agents-shipgate trigger --base … --head … --json. Both return should_run: true, run_shipgate: true, skip_reason: null, dry_run_recommended: false.

  • All four acceptance criteria from the issue confirmed by hand, including the ADK quickstart spelling (the Agent alias), the docs-only google-adk mention, FunctionTool( / LongRunningFunctionTool( retention, and a real dependency bump still routing dry_run.

  • Full suite green (pytest tests/ exit 0), python -m ruff check . clean, docs/triggers.json parses.

  • llms-full.txt regenerated with python scripts/build-llms-full.py (AGENTS.md is its source).

One finding worth a reviewer's attention

The first draft of the ADK rule also watched the spaced form tools = [. tests/test_codex_boundary_check.py caught it: diff_contains is a plain substring match, so that token also swallows enabled_tools = [...] in a Codex TOML config, and the token started appearing in diff_tokens on unrelated boundary checks. That is a smaller instance of exactly the defect this PR is fixing — a token with no structural meaning being published as if it were structure — so the spaced form is deliberately not matched. The reasoning is recorded in the rule's own rationale and pinned by test_triggers_do_not_watch_the_spaced_toml_tools_array_token.

Consequence of dropping it: an ADK agent that builds its list separately (tools = [...] then Agent(..., tools=tools)) is not matched by this rule. That is the conservative side of the trade, and the issue scopes the fix to tools=[...].

Coverage

8 new tests in tests/test_public_surface_contract.py, alongside the existing evaluator contract tests:

  • the #6605 shape routes run_shipgate and does not match the dependency rule;
  • the quickstart Agent alias routes the same way;
  • FunctionTool( / LongRunningFunctionTool( still carried by TRIGGER-FUNCTION-TOOL-DECORATOR;
  • a docs-only google-adk mention is not classified as a version bump (and still reports the token in diff_tokens, which draws no conclusion);
  • the dependency rule needs a manifest — verified across pyproject.toml, requirements-dev.txt, services/api/package.json, uv.lock;
  • a structural pin that the dependency rule's when keeps a path leg, so reverting it to a bare any_of of tokens fails CI;
  • the TOML-token negative control above.

Note for the reviewer

This PR changes docs/triggers.json, so Shipgate's own gate will raise SHIP-VERIFY-TRIGGER-CATALOG-DRIFT (medium → human review). That is the check working as designed — it exists so a human confirms a catalog edit does not carve out a gate-evasion path. The two rule changes here move in opposite directions on that axis: one adds a positive route that did not exist, and the other narrows an advisory rule so it stops claiming evidence it does not have. Neither removes a run_shipgate route.

Release-readiness notes

  • No user-code import added to default scan paths — the new rule is diff_contains / glob predicates only; nothing imports, executes, or fetches
  • No network access added to default scan paths
  • New or changed check IDs are documented in docs/checks.md — n/a, no check IDs added or changed
  • Report/schema changes are additive or documented in STABILITY.md — n/a, no report or schema change; catalog schema_version stays 0.3 and every rule ID is preserved

🤖 Generated with Claude Code

@pengfei-threemoonslab
pengfei-threemoonslab force-pushed the claude/github-issue-315-df057d branch from 8f53aba to 145f083 Compare August 8, 2026 02:03

@pengfei-threemoonslab pengfei-threemoonslab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review findings (not an approval): I found two reproducible routing regressions that should be fixed before merge. The existing-list ADK case still fails open, and the new dependency-manifest allowlist removes advisory coverage for common real bump surfaces. CI and the public-surface tests are green, but neither adversarial case is covered. Shipgate's expected trust-root review remains separate: the exact-ref verifier reports human_review_required / review_required with 0 blockers and 4 review items.

Comment thread docs/triggers.json Outdated
"when": {
"all_of": [
{"any_of": [
{"diff_contains": "google.adk"},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Existing ADK tool-list edits still bypass this trigger. This leg requires google.adk/google-adk to occur in the unified diff, but an ordinary edit adding a tool to an existing LlmAgent(..., tools=[...]) list does not repeat the distant import. On this head, a hunk containing LlmAgent(, tools=[, and +delete_account returns should_run=false with skip_reason="no_match". That contradicts the published “Adds/changes … tools list | Yes” contract and leaves the common modified-list case unfixed. Please derive ADK context from the head file/existing detector (or another source-aware signal) and add a regression test that modifies an existing list rather than adding the whole file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 735d003. Reproduced your exact case first — a hunk with LlmAgent(, tools=[, and +delete_account returned should_run=false, skip_reason="no_match" on the previous head.

You identified the root cause precisely: git diff carries three lines of context, so the constructor and the list are in the hunk and the import is dozens of lines outside it. Requiring the import meant the rule covered whole-file additions and nothing else — which does contradict the "Adds/changes" row I published.

The context leg is now a disjunction:

{"all_of": [
  {"diff_contains": "tools=["},
  {"any_of": [
    {"diff_contains": "LlmAgent("},
    {"all_of": [
      {"any_of": [{"diff_contains": "google.adk"}, {"diff_contains": "google-adk"}]},
      {"diff_contains": "Agent("}
    ]}
  ]}
]}

LlmAgent( carries the modified-list case on its own — no other supported framework exports that class name, so it is an ADK-identifying signal that survives outside the import hunk. Regression test test_triggers_google_adk_modified_tools_list_routes_run uses a real modified-list hunk (-tools=[lookup_account] / +tools=[lookup_account, delete_account]) rather than a whole-file addition.

On deriving context from the head file. I did not take that route, and want to be explicit about why rather than have it look like an oversight. The catalog's contract is that an agent applies it to a PR diff; the vocabulary has no predicate that reads head-tree content (file_present is real vocabulary, but the evaluator implements it only for shipgate.yaml, keyed off manifest_present). Generalizing it means new predicate semantics and a schema_version bump, which every external consumer that pre-fetched triggers.json would then have to implement — the surface expansion #315's own scoping ruled out. LlmAgent( is a source-aware signal that closes the reported case inside the existing vocabulary.

Residual gap, stated rather than hidden. A modified list on the Agent alias (Agent(..., tools=[...]) with no ADK token in the hunk) is still unmatched. It was also unmatched on the base catalog, so it is not a regression, but it is real. I deliberately did not close it by widening: Agent( + tools=[ is indistinguishable from CrewAI's Agent(role=..., tools=[...]) by diff text alone, and routing CrewAI under a rule ID naming Google ADK would be precisely the "states a conclusion its evidence does not support" defect this PR exists to fix. test_triggers_bare_agent_call_without_adk_context_does_not_claim_adk pins that boundary. The gap is documented in the rule rationale and in the AGENTS.md trigger section.

Closing it properly needs the head-file signal you suggested. Happy to open a follow-up issue for a source-aware predicate (catalog schema_version bump, head-tree read, external-consumer contract) if you want that as its own change.

Comment thread docs/triggers.json
{"glob": "**/pyproject.toml"},
{"glob": "**/setup.py"},
{"glob": "**/setup.cfg"},
{"glob": "**/requirements*.txt"},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] This closed manifest allowlist regresses real framework bumps. Common pip-tools inputs such as requirements.in and constraints.in are omitted. For requirements.in with google-adk==1.0.0 changed to 1.1.0, the base catalog sets dry_run_recommended=true, while this head returns skip_reason="no_match" and dry_run_recommended=false. That contradicts the CHANGELOG claim that real-bump coverage is unchanged. Please include requirements*.in/constraints*.in (and review modern lock formats such as pylock.toml/pdm.lock), preferably via a shared canonical manifest set, with regression fixtures.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 735d003. Reproduced exactly as you described — requirements.in with google-adk==1.0.01.1.0:

catalog verdict
base (origin/main) dry_run_recommended: true, TRIGGER-FRAMEWORK-VERSION-BUMP
previous head skip_reason: "no_match", dry_run_recommended: false
now dry_run_recommended: true, TRIGGER-FRAMEWORK-VERSION-BUMP

Same for constraints.in, pylock.toml, and pdm.lock. You are right that this contradicted the CHANGELOG claim; that sentence was false and is corrected — real-bump coverage is now wider than before this PR, not "unchanged". The failure mode was exactly what you named: a closed allowlist written from memory, which silently regressed every pip-tools repository, where the bump is authored in .in and compiled to .txt.

Taken the shared-canonical-set route you preferred. New DEPENDENCY_MANIFEST_GLOBS in src/agents_shipgate/core/dependency_manifests.py is now the single answer to "is this a file where a dependency version is declared or locked?", with is_dependency_manifest(path) alongside it. docs/triggers.json projects the tuple, and test_dependency_manifest_projection_matches_runtime_set asserts the catalog's glob leg equals the constant element-for-element — the same drift guard boundary_adapters already gets from test_trigger_boundary_adapter_projection_matches_runtime_registry. Adding a manifest in one place without the other now fails CI.

Coverage added beyond the four you flagged, keeping the "declares or locks a version" rule as the membership test:

  • pip-tools, both halvesrequirements*.in, requirements/*.in, constraints*.in (plus the existing .txt forms)
  • modern Python lockspdm.lock, PEP 751 pylock.toml and pylock.*.toml (the spec permits the named form)
  • condaenvironment.yaml and conda-lock.{yml,yaml} alongside the existing environment.yml
  • Nodenpm-shrinkwrap.json, bun.lock, bun.lockb alongside npm/pnpm/yarn
  • JVMgradle/libs.versions.toml version catalogs alongside pom.xml and the Gradle build files, since conductor-oss is a JVM token

Regression fixtures: _DEPENDENCY_MANIFEST_SAMPLE_PATHS holds one representative path per glob (34 paths, including nested spellings like services/api/pyproject.toml and requirements/base.in). Each is parametrized through test_dependency_manifest_samples_are_recognized_by_both, asserting both that is_dependency_manifest() matches it and that a framework bump there routes dry_run_recommended: true — so a glob that reads plausibly but matches nothing fails loudly instead of leaving a silent hole in the direction you found. test_non_manifest_paths_are_not_dependency_manifests is the negative control (README.md, Dockerfile, src/agent.py stay out: a file that merely mentions a package cannot support the claim that a dependency changed).

Full suite green (exit 0) and ruff check . clean on the rebased tree.

pengfei-threemoonslab and others added 2 commits August 8, 2026 01:01
…ange

An ADK sample whose root agent is `LlmAgent(name="support_agent",
tools=[lookup_account, delete_account])` routed as `dry_run_only`. The
only rule that fired was `TRIGGER-FRAMEWORK-VERSION-BUMP`, on a raw
`google-adk` string in the diff, and it reported the result as a
framework upgrade.

Two defects sat behind that.

Catalog drift, not a missing capability: detection
(`GOOGLE_ADK_AGENT_CLASSES`), the ADK adapter, and the binding graph all
resolve this shape already — `docs/triggers.json` was the one component
with no ADK rule. Plain functions passed to `tools=[...]` carry no
decorator, so `@function_tool` / `FunctionTool(` never sees them.
`TRIGGER-GOOGLE-ADK-AGENT-TOOLS-CHANGED` conjoins a `google.adk` module
path, an `Agent(` / `LlmAgent(` construction, and a `tools=[...]`
argument — static, and drawn from the engine's own signal vocabulary.

A rule stating a conclusion its evidence cannot support: nothing about
the string `google-adk` establishes that a version moved. It comes just
as easily from install prose or a sample import, which is why a
docs-only change could be classified as a framework upgrade.
`TRIGGER-FRAMEWORK-VERSION-BUMP` now needs both halves of its claim —
the package token AND a changed dependency manifest — and its rationale
reports the co-occurrence it observed rather than the upgrade it
inferred. Real bumps are unaffected.

The spaced form `tools = [` is deliberately not watched: `diff_contains`
is a substring match, so it also swallows `enabled_tools = [...]` in a
Codex TOML config — a token with no structural meaning, which is the
reporting defect this change is fixing.

Rule IDs and the catalog `schema_version` (0.3) are unchanged; this is
rule precision inside the existing schema, so a pre-fetched catalog
keeps working.

Fixes #315

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…p routing

Addresses both PR review findings, each reproduced first.

[P1] An edit adding a tool to an existing `LlmAgent(..., tools=[...])`
returned `no_match`. The rule required a `google.adk` token in the diff,
but `git diff` shows three lines of context — the constructor and the
list are in the hunk, the import is dozens of lines away and is not. So
the rule covered whole-file additions only, and contradicted the
published "Adds/changes ... tools list | Yes" row for every subsequent
edit to the same list.

The context leg is now a disjunction: `LlmAgent(` on its own, since no
other supported framework exports that class name, OR a `google.adk`
module path together with `Agent(`. `Agent(` stays gated because CrewAI
constructs `Agent(..., tools=[...])` too, and routing that under a rule
ID naming Google ADK would be the same overstatement this PR is fixing.
Residual gap, documented in the rule and AGENTS.md: a *modified* list on
the `Agent` alias, which diff text alone cannot attribute to ADK.

[P2] The hand-written manifest allowlist dropped real bumps. pip-tools
repositories author a bump in `requirements.in` and compile it to
`requirements.txt`; `google-adk==1.0.0` -> `1.1.0` in a `.in` file went
from `dry_run_recommended: true` on the base catalog to `no_match` here.
`pylock.toml` and `pdm.lock` were missing for the same reason: a closed
list written from memory.

The set now lives in one place — `DEPENDENCY_MANIFEST_GLOBS` in
`core/dependency_manifests.py` — covering both halves of the pip-tools
pair, the modern lockfiles, and Node/JVM manifests for the JS and
`conductor-oss` tokens. `docs/triggers.json` projects it, and a contract
test pins the projection against the constant, the same drift guard
`boundary_adapters` already had. Real-bump coverage is now wider than
before this PR, not merely preserved; the CHANGELOG claim that it was
"unchanged" was false and is corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pengfei-threemoonslab
pengfei-threemoonslab force-pushed the claude/github-issue-315-df057d branch from 145f083 to 735d003 Compare August 8, 2026 08:12
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

Both findings were reproducible as written, and both are fixed in 735d003. I reproduced each against the base catalog before changing anything, so the fix and the regression both have evidence.

[P1] Modified ADK tool lists. Root cause was exactly as you diagnosed: the rule demanded a google.adk token in the diff, and a three-line context hunk around an existing agent doesn't carry the import — so the rule covered whole-file additions only. The context leg is now LlmAgent( on its own (no other supported framework exports that class name) or google.adk + Agent(. Your repro now returns should_run=true.

[P2] Dependency-manifest allowlist. A genuine regression against base — requirements.in went dry_run_recommended: trueno_match. Moved to a shared DEPENDENCY_MANIFEST_GLOBS in core/dependency_manifests.py, projected into the catalog under a parity test, and widened to cover both halves of the pip-tools pair, pdm.lock, PEP 751 pylock*.toml, conda locks, and Node/JVM manifests. The CHANGELOG sentence claiming coverage was "unchanged" was false and now says what is true: coverage is wider than before this PR.

case base previous head now
modified LlmAgent tools list no_match no_match run_shipgate
requirements.in bump dry_run no_match dry_run
constraints.in bump dry_run no_match dry_run
pylock.toml bump dry_run no_match dry_run
pdm.lock bump dry_run no_match dry_run
original #315 repro (real google/adk-python#6605) dry_run_only run_shipgate run_shipgate

One residual gap I want on the record rather than buried. A modified tools list on the Agent alias with no ADK token in the hunk is still unmatched (it was unmatched on base too, so not a regression). I chose not to close it by widening, because Agent( + tools=[ cannot be told apart from CrewAI's Agent(role=..., tools=[...]) by diff text alone — routing CrewAI under a rule ID naming Google ADK would reproduce the exact defect this PR fixes. Closing it properly needs the head-file signal you suggested, which is a predicate-vocabulary addition and a catalog schema_version bump. Say the word and I'll open a follow-up issue for it; the gap is documented in the rule rationale and the AGENTS.md trigger section in the meantime.

Also rebased onto c07ba35c (#332 landed mid-review). Full suite exit 0, ruff check . clean, 12 new tests on the trigger contract.

On the trust-root review: expected and correct — the PR edits docs/triggers.json, which is what SHIP-VERIFY-TRIGGER-CATALOG-DRIFT exists to route to a human. Worth noting for that review that neither change removes a run_shipgate route: P1 adds one that did not exist, and the dependency change narrows an advisory rule while widening the manifest set it reads.

@pengfei-threemoonslab
pengfei-threemoonslab merged commit 46f83c6 into main Aug 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(trigger): recognize Google ADK LlmAgent tools=[...] changes as tool-surface changes

1 participant