Skip to content

[TEST — DO NOT MERGE] Demo for #932 secret-coherence pipeline#935

Closed
jottakka wants to merge 1 commit intofeat/toolkit-docs-secret-coherencefrom
test/secret-coherence-demo
Closed

[TEST — DO NOT MERGE] Demo for #932 secret-coherence pipeline#935
jottakka wants to merge 1 commit intofeat/toolkit-docs-secret-coherencefrom
test/secret-coherence-demo

Conversation

@jottakka
Copy link
Copy Markdown
Contributor

@jottakka jottakka commented Apr 18, 2026 •
edited
Loading

⚠️ Testing-only PR — do not merge

Base branch is #932, not main. Close this PR after testing; the phantom secret must not land on main.

What changed compared to the first test run

The first trigger on this branch (#936) produced no cleanup because --skip-unchanged short-circuits before enforceSecretCoherence runs when the Engine API's summary endpoint reports no version change for github. The phantom secret diff was never evaluated.

#932 now has a workflow_dispatch.inputs.providers input that bypasses --skip-unchanged and --all for manual runs. This branch is rebased on that change.

How to run it (updated)

  1. Confirm ANTHROPIC_API_KEY is set as a repo secret. Without it the scanner runs but the LLM edit is skipped.
  2. Go to Actions → Generate toolkit docs → Run workflow.
  3. Pick branch test/secret-coherence-demo.
  4. In the providers input, enter: Github.
  5. Click Run workflow.

The run now uses --providers Github without --skip-unchanged, so the generator processes Github end-to-end — the scanner diffs the committed JSON (with the phantom) against fresh Engine API data (without it), identifies GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN as removed, scans chunks, and (if the Anthropic key is present) drives Sonnet 4.6 to minimally edit the stale references out.

What you should see in the AUTO PR

In the resulting [AUTO] PR's diff on toolkit-docs-generator/data/toolkits/github.json:

  • ✓ Phantom secret removed from Github.AssignPullRequestUser .secrets / .secretsInfo (that happens automatically because Engine API doesn't return it).
  • ✓ The custom_section chunk (the Secrets Setup section) drops the row / note lines that reference GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN, leaving the GITHUB_SERVER_URL row intact.
  • ✓ Unrelated chunks (Enterprise Support callout, App Permissions Summary, Configuration & Setup) stay byte-identical — confirms the edit is minimal.
  • ✓ The run log should include a warning like Stale secret reference in toolkit_chunk #N: GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN.

When done

Close this PR without merging. Delete the branch.

Refs #932

@vercel
Copy link
Copy Markdown

vercel bot commented Apr 18, 2026 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Apr 18, 2026 7:55pm

Request Review

@jottakka jottakka self-assigned this Apr 18, 2026
jottakka added a commit that referenced this pull request Apr 18, 2026
@jottakka @claude
feat(workflow): Node 24 opt-in + focused workflow_dispatch for manual…
8c0d5ae 
… runs

Two workflow additions driven by PR #936 feedback:

1. Job-level `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"` opts all
   JavaScript actions into Node 24 ahead of the 2026-06-02
   deprecation. actions/checkout@v4, actions/setup-node@v4,
   peter-evans/create-pull-request@v7, and pnpm/action-setup@v4 all
   trigger the "Node.js 20 actions are deprecated" annotation today;
   the opt-in silences it and matches the runtime we'll be forced
   onto anyway.

2. New `workflow_dispatch` input `providers`. When set to a
   comma-separated provider list (e.g. "Github"), the run uses
   `--providers "$providers"` AND drops `--skip-unchanged` so the
   secret-coherence scan actually re-evaluates those toolkits — even
   when the Engine API reports no version change. Scheduled and
   porter_deploy_succeeded runs keep the previous `--all
   --skip-unchanged` behavior. This is what lets the #935 demo PR
   actually exercise the pipeline end-to-end: trigger the workflow
   with `providers=Github` and the phantom secret gets surfaced +
   cleaned.

Tests added: workflow assertions for the new env var and the
providers input fallback structure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jottakka jottakka force-pushed the test/secret-coherence-demo branch from 5638ef3 to ac00598 Compare April 18, 2026 19:17
jottakka added a commit that referenced this pull request Apr 18, 2026
@jottakka @claude
feat(cli): surface per-toolkit merge warnings to stdout in --all runs
ad7b8c8 
The stale-secret scanner, coverage-gap detector, and summary-generation
failures all push warnings onto `result.warnings`. Per-provider mode
already echoes those to stdout (line 848 of cli/index.ts). The --all
and regenerate-all paths did not — they only appended to the run log
file on disk, which GitHub Actions runs don't expose.

Result: on the #935 demo, the workflow ran, the phantom secret was
removed from the tool's .secrets array, but no cleanup was applied to
the stale doc chunk that still referenced it AND there was no signal
in the CI log explaining why. The warnings that would have explained
"stale secret detected but edit failed" or "stale secret detected but
no editor configured" were present in memory but discarded.

This commit prints every non-empty `mergeResult.warnings` to stdout
right after `mergeAllToolkits()` returns, in both the `generate --all`
and `regenerate --all` paths. Format matches existing spinner output:

    ⚠ Github: 2 warning(s)
      - Stale secret reference in toolkit_chunk #4: GITHUB_CLASSIC_...
      - Secret cleanup edit failed for Github (documentation_chunk): ...

551 tests pass, type-check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jottakka added a commit that referenced this pull request Apr 18, 2026
@jottakka @claude
feat(cli): surface per-toolkit merge warnings to stdout in --all runs
e6381ed 
The stale-secret scanner, coverage-gap detector, and summary-generation
failures all push warnings onto `result.warnings`. Per-provider mode
already echoes those to stdout (line 848 of cli/index.ts). The --all
and regenerate-all paths did not — they only appended to the run log
file on disk, which GitHub Actions runs don't expose.

Result: on the #935 demo, the workflow ran, the phantom secret was
removed from the tool's .secrets array, but no cleanup was applied to
the stale doc chunk that still referenced it AND there was no signal
in the CI log explaining why. The warnings that would have explained
"stale secret detected but edit failed" or "stale secret detected but
no editor configured" were present in memory but discarded.

This commit prints every non-empty `mergeResult.warnings` to stdout
right after `mergeAllToolkits()` returns, in both the `generate --all`
and `regenerate --all` paths. Format matches existing spinner output:

    ⚠ Github: 2 warning(s)
      - Stale secret reference in toolkit_chunk #4: GITHUB_CLASSIC_...
      - Secret cleanup edit failed for Github (documentation_chunk): ...

551 tests pass, type-check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jottakka @claude
test(toolkit-docs-generator): DO NOT MERGE — demo for #932 secret coh…
c58424f 
…erence

Seeds a phantom secret (GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN) into
Github.AssignPullRequestUser's secrets and secretsInfo so that a fresh
workflow run against the live Engine API identifies the secret as
"removed" when it compares the committed toolkit against the generator
output.

The existing documentation chunk in github.json still references this
same secret name (real-world residue from when the notification tools
were dropped in #922), so the secret-coherence step from #932 should:

1. Detect `GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN` as removed on the
   `Github.AssignPullRequestUser` tool.
2. Scan chunks and find the matching text in the `custom_section` chunk.
3. Call the Claude Sonnet 4.6 editor to minimally edit that chunk —
   deleting the table row and note lines that reference the removed
   secret while preserving the rest of the setup documentation.

To test: trigger `Generate toolkit docs` workflow via workflow_dispatch
on this branch. Confirm the resulting AUTO PR's diff on github.json:
- Removes the phantom secret from the tool's .secrets / .secretsInfo.
- Rewrites the Secrets Setup chunk to drop the stale rows.
- Leaves unrelated chunk content intact (no re-summarization).

Do not merge this commit. The phantom secret must not land on main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jottakka jottakka force-pushed the test/secret-coherence-demo branch from ad7b8c8 to c58424f Compare April 18, 2026 19:53
@jottakka jottakka closed this Apr 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@jottakka jottakka

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jottakka