Skip to content

test(source-context): cover partial-label and mismatched promotion markers - #2866

Merged
stranske merged 2 commits into
mainfrom
closer/followup-2865-promotion-trust-coverage
Aug 1, 2026
Merged

test(source-context): cover partial-label and mismatched promotion markers#2866
stranske merged 2 commits into
mainfrom
closer/followup-2865-promotion-trust-coverage

Conversation

@stranske

@stranske stranske commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Source: Issue #2865

Closes #2865

Automated Status Summary

Scope

Trusted dependency-repair promotion metadata must suppress incidental issue-link routing, while forged or incomplete markers must preserve normal issue routing.

Context for Agent

Related Issues/PRs

Tasks

  • Keep promotion handling consistent in the root and consumer-template source resolvers.
  • Validate trusted provenance before suppressing issue routing.
  • Cover trusted and forged markers in the source-context tests.
  • Resolve the current template-parity review finding.

Acceptance criteria

  • Trusted promotion markers suppress incidental issue links only when provenance validates.
  • Forged or mismatched markers preserve the issue reference and source type.
  • Root and template implementations remain synchronized and tests pass.

Summary by CodeRabbit

  • Tests
    • Added coverage for dependency-repair promotion trust handling.
    • Verified that incomplete labels and invalid promotion data preserve issue routing.
    • Confirmed consistent repository resolution for trusted and untrusted label combinations.

…rkers

The #2862 verifier comparison flagged that promotion trust was only
exercised for the both-labels and no-labels paths, so a weakened gate
could suppress issue routing undetected. Adds partial-label, mismatched
payload, and root/consumer-template parity coverage.
Copilot AI review requested due to automatic review settings August 1, 2026 00:47
@cursor

cursor Bot commented Aug 1, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Aug 1, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The source-context tests now cover incomplete and invalid dependency-repair promotion metadata. They also compare the primary and consumer-repository resolvers across trusted, partial, and absent promotion-label states.

Changes

Promotion trust routing

Layer / File(s) Summary
Resolver trust and parity coverage
.github/scripts/__tests__/source-context.test.js
Tests preserve GitHub issue routing for incomplete or invalid promotion metadata and verify parity between the primary and consumer-repository resolvers.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested labels: needs-human, agent:needs-attention

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The tests cover trusted and forged markers and resolver parity, but the two stated review items remain unresolved. Assert context.sourceRef in malformed or forged-marker tests and route malformed payload cases through the consumer resolver.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies tests for partial-label and mismatched promotion markers.
Out of Scope Changes check ✅ Passed The changes add source-context tests that directly support the linked issue objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch closer/followup-2865-promotion-trust-coverage

Comment @coderabbitai help to get the list of available commands.

@agents-workflows-bot

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: 7b84e60
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 0

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds targeted test coverage for dependency-repair promotion trust boundaries in the PR source-context resolver, focusing on the previously untested “partial label” and “trusted labels + bad marker payload” paths, plus a parity check to prevent root/template behavior drift.

Changes:

  • Adds tests for partial promotion-label combinations to ensure issue routing is not suppressed unless both trust labels are present.
  • Adds tests for malformed/mismatched promotion payloads even when both trust labels are present, ensuring issue routing is preserved.
  • Adds a root vs templates/consumer-repo resolver deep-equality assertion across key label combinations to guard against behavioral drift.

Comment thread .github/scripts/__tests__/source-context.test.js

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f0ae031947

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/__tests__/source-context.test.js
@stranske stranske added agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation agent:retry Add to trigger agent retry after rate limit or pause follow-up labels Aug 1, 2026
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #2866. Do not edit.

@agents-workflows-bot

agents-workflows-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

🤖 Bot Comment Handler

  • Agent: codex
  • Bot comments to address: 3

The agent has been assigned to this PR to address the bot review comments.

Instructions for agent

  1. Implement suggested fixes that improve the code
  2. Skip suggestions that don't apply (note why in your response)

The bot comment handler workflow has prepared context in the artifacts.

@stranske

stranske commented Aug 1, 2026

Copy link
Copy Markdown
Owner Author

Opener lane: routing repaired, but keepalive cannot be dispatched (owner action required)

Routing repair applied this round. This PR was opened with zero labels, so hasAutomationSignal() in .github/scripts/keepalive_orchestrator_gate_runner.js:64-74 returned false and the keepalive gate self-heal never ran — the Dispatch keepalive orchestrator job skipped on every PR event. Applied agent:codex, agents:keepalive, autofix, agent:retry, follow-up, matching the convention of the last labelled closer follow-up (#2097). Branch prefix closer/ is not a registry prefix, but that is fine: the gate runner resolves the agent from the concrete agent:* label first and only falls back to branch inference (inferAgentFromBranch, same file lines 42-62).

Repair confirmed effective, then blocked by a repo-level approval gate. The label change did trigger Agents Keepalive Loop (it is no longer invisible), but every resulting run completes as action_required with zero jobs — as does an explicit workflow_dispatch with pr_number=2866 force_retry=true (run 30677515717).

Evidence gathered, with alternatives ruled out:

  • Regression boundary: the last successful agents-keepalive-loop.yml run was 2026-07-31T19:37:32Z. Every one of the 44 runs since is action_required, across pull_request, workflow_run, and workflow_dispatch triggers.
  • Not a fork/contributor gate: the triggering actor is the repo owner (stranske), and POST /actions/runs/30677373863/approve refuses with "This run is not from a fork pull request or queued by the Actions bot" (403).
  • Not an environment gate: agent-standard, agent-high-privilege, and copilot all report zero protection_rules, and /pending_deployments for the blocked run is empty.
  • Not a repo Actions setting visible via API: actions/permissions = {enabled: true, allowed_actions: "all"}, permissions/workflow = {default_workflow_permissions: "write"}.
  • Scoped to a specific workflow set, not the repo: Agents Keepalive Loop, Agents Auto-Pilot, Create Issue from Verification, and Create New PR from Verification are all blocked, while Agents PR meta manager, Agents Verifier, Health 45 Agents Guard, CI Autofix Loop, and Gate run normally on this same head. Consumer repos are healthy apart from single Agents PR Event Hub runs in the same state.
  • Coincident with main commit 0ed299f1 (chore(deps): update actions/setup-python action to v7, chore(deps): update actions/setup-python action to v7 #2795) at 2026-07-31T19:37:28Z, four seconds before the last good run. That commit touched all four blocked workflows — but it also touched pr-00-gate.yml, which still runs, so it is a correlation to check rather than a proven cause.

Owner decision required: approve/enable these runs from the Actions UI (they should show Approve and run), or identify the account/org-level policy that changed at ~19:37Z on 2026-07-31. This is the same condition already recorded against #2863 (PR #2810) and #2864 (PR #2826), so it is now blocking three PRs.

Next lane owner: closer. This PR is non-draft, MERGEABLE, and all required product/Gate contexts pass. Because keepalive cannot be dispatched until the gate above clears, the two open review threads will not be addressed automatically:

  1. copilot-pull-request-reviewer — assert context.sourceRef in the malformed/forged-marker test.
  2. chatgpt-codex-connector (P2) — exercise the malformed-payload cases through the consumer resolver.

The closer should either resolve those two threads directly and merge, or merge and carry them as a bounded follow-up. No opener action remains.

Posted by the opener lane (cursor) — routing repair + blocker evidence only; no code changes to this branch.

@stranske

stranske commented Aug 1, 2026

Copy link
Copy Markdown
Owner Author

Addressed both actionable review findings in 39b3d7a. The malformed-payload cases now assert sourceRef === "#99" and run through the consumer-template resolver with the full expected issue context. Focused validation: node --test .github/scripts/__tests__/source-context.test.js (36 passing).

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/__tests__/source-context.test.js:
- Around line 351-381: Update the parity assertion in the consumer template
resolver synchronization test to use assert.deepStrictEqual instead of
assert.deepEqual, ensuring templateResolvePrSourceContext and
resolvePrSourceContext match exactly, including primitive types.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 70f77b99-b717-40b4-a8fd-98cc400363a9

📥 Commits

Reviewing files that changed from the base of the PR and between f2d406b and 39b3d7a.

📒 Files selected for processing (1)
  • .github/scripts/__tests__/source-context.test.js

Comment thread .github/scripts/__tests__/source-context.test.js
@stranske
stranske merged commit dbce5dd into main Aug 1, 2026
48 checks passed
@stranske
stranske deleted the closer/followup-2865-promotion-trust-coverage branch August 1, 2026 01:38
@stranske stranske added the verify:compare Compare multiple LLM evaluations label Aug 1, 2026
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.6-terra PASS 88% The change adds focused source-context coverage for the acceptance scenarios. It verifies that incomplete label sets do not suppress an incidental issue reference, that malformed/mismatched promoti...
anthropic claude-sonnet-5 PASS 72% This PR is a pure test-addition change (122 lines added, 0 removed) to .github/scripts/tests/source-context.test.js. It adds targeted test cases for: (1) partial promotion label sets that shoul...
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.6-terra
  • Verdict: PASS
  • Confidence: 88%
  • Scores:
    • Correctness: 9.0/10
    • Completeness: 9.0/10
    • Quality: 9.0/10
    • Testing: 9.0/10
    • Risks: 9.0/10
  • Summary: The change adds focused source-context coverage for the acceptance scenarios. It verifies that incomplete label sets do not suppress an incidental issue reference, that malformed/mismatched promotion payloads remain untrusted even with both promotion labels, and that the consumer-template resolver produces behavior consistent with the root resolver across trusted and incomplete-label cases. The tests assert preservation of GitHub-issue source type, issue number, source reference, and issue requirement for untrusted inputs. The test structure is readable and uses representative promotion metadata without introducing production-code, compatibility, security, or performance risk.

anthropic

  • Model: claude-sonnet-5
  • Verdict: PASS
  • Confidence: 72%
  • Scores:
    • Correctness: 8.0/10
    • Completeness: 7.0/10
    • Quality: 8.0/10
    • Testing: 9.0/10
    • Risks: 8.0/10
  • Summary: This PR is a pure test-addition change (122 lines added, 0 removed) to .github/scripts/tests/source-context.test.js. It adds targeted test cases for: (1) partial promotion label sets that should NOT suppress issue routing, (2) mismatched/forged promotion payloads (zero PR number, invalid SHA format, missing promotion_base_sha) that should preserve issue routing in both the root and consumer-template resolvers, and (3) a template-vs-root behavioral parity check across multiple label combinations. All CI workflows (pr-00-gate, pr-11-ci-smoke, selftest-ci) report success post-merge, confirming these new tests pass against the existing implementation. The tests directly exercise the acceptance criteria: trusted markers only suppress issue links when full provenance (matching PR, valid SHAs, base SHA) validates, and forged/incomplete markers preserve issue routing — verified identically in both root and template resolvers. Since the diff contains no implementation changes, correctness/completeness rely on the assumption that the underlying resolver logic was already implemented correctly in the codebase (supported by all tests passing); the PR title and scope explicitly frame this as a test-coverage closure task. Given green CI and well-targeted edge-case coverage aligned with the stated acceptance criteria, this qualifies as PASS, though full confidence would require confirming the resolver implementation itself already meets the criteria — which the passing tests substantiate but the diff itself does not show directly.
  • Concerns:
    • The diff is test-only (+122/-0); no changes to source_context.js or the consumer-template resolver are included in this PR, so the 'keep resolvers synchronized' and 'validate trusted provenance' implementation tasks are not directly verifiable from this diff alone — they must already exist in the codebase from prior work.
    • Acceptance criteria phrasing implies both implementation and test coverage; this PR appears to only close the testing gap, assuming the underlying resolver logic (parseDependencyRepairPromotionSource, resolvePrSourceContext, template resolver) was already correct.
    • No mutation/negative test around malformed JSON parsing edge cases beyond the ones shown, though the added scenarios (partial labels, mismatched SHAs, missing promotion_base_sha) are reasonably comprehensive.

Agreement

  • Verdict: PASS (all providers)
  • Correctness: scores within 1 point (avg 8.5/10, range 8.0-9.0)
  • Quality: scores within 1 point (avg 8.5/10, range 8.0-9.0)
  • Testing: scores within 1 point (avg 9.0/10, range 9.0-9.0)
  • Risks: scores within 1 point (avg 8.5/10, range 8.0-9.0)

Disagreement

Dimension openai anthropic
Completeness 9.0/10 7.0/10

Unique Insights

  • openai: The change adds focused source-context coverage for the acceptance scenarios. It verifies that incomplete label sets do not suppress an incidental issue reference, that malformed/mismatched promotion payloads remain untrusted even with both promotion labels, and that the consumer-template resolve...
  • anthropic: The diff is test-only (+122/-0); no changes to source_context.js or the consumer-template resolver are included in this PR, so the 'keep resolvers synchronized' and 'validate trusted provenance' implementation tasks are not directly verifiable from this diff alone — they must already exist in the codebase from prior work.; Acceptance criteria phrasing implies both implementation and test coverage; this PR appears to only close the testing gap, assuming the underlying resolver logic (parseDependencyRepairPromotionSource, resolvePrSourceContext, template resolver) was already correct.; No mutation/negative test around malformed JSON parsing edge cases beyond the ones shown, though the added scenarios (partial labels, mismatched SHAs, missing promotion_base_sha) are reasonably comprehensive.

🔍 LangSmith Traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:codex Agent-created issues from Codex agent:retry Add to trigger agent retry after rate limit or pause agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation follow-up verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authenticate dependency-repair promotion provenance for source routing

3 participants