feat(onboarding): template-driven scaffold + register prompts/eval in matrix#141
Closed
arnaudlh wants to merge 6 commits into
Closed
feat(onboarding): template-driven scaffold + register prompts/eval in matrix#141arnaudlh wants to merge 6 commits into
arnaudlh wants to merge 6 commits into
Conversation
The waza model catalog now ships gpt-5-codex under its versioned ID gpt-5.3-codex. Align manifest tiers and bench-prompt argument hints so dispatched runs resolve to a valid model. - .github/evals/manifest.yaml: pilot + expanded tier model lists - .github/prompts/agent-bench.prompt.md: default models in argument-hint + body - .github/prompts/skill-bench.prompt.md: default models in argument-hint + body 🔖 - Generated by Copilot
Rewrite git-ape-onboarding as a skill-driven CLI playbook backed by a
sync-able template bundle. The previous .exampleyml workflows lived in
this repo's .github/workflows/ and were copy-pasted by users; they're
now first-class templates under the skill and pushed into target repos
by scripts/sync-templates.{sh,ps1}.
What ships:
- .github/agents/git-ape-onboarding.agent.md: rewritten flow + tools
- .github/skills/git-ape-onboarding/SKILL.md: new playbook structure
- .github/skills/git-ape-onboarding/scripts/: bash + pwsh helpers
- scaffold-repo.{sh,ps1}: bootstrap target repo
- sync-templates.{sh,ps1}: drop-in workflow + instructions update
- .github/skills/git-ape-onboarding/templates/: canonical target-repo
artifacts (copilot-instructions.md, workflows/git-ape-{plan,deploy,
destroy,verify,drift}.yml + drift agentic workflow + drift lockfile)
- .github/evals/git-ape-onboarding/: positive + negative tasks for
first-time-setup, multi-env, skip-on-collision, and storage refusal
- .github/workflows/git-ape-onboarding-template-check.yml: CI check
that the shipped templates pass actionlint and round-trip cleanly
- .github/evals/manifest.yaml: register git-ape-onboarding in pilot
tier (matches its prior 4-model bench coverage)
Removed:
- .github/workflows/git-ape-{deploy,destroy,plan,verify}.exampleyml:
retired — their content is now in skills/.../templates/workflows/
The .exampleyml extension was a workaround to keep GitHub Actions from
auto-loading workflow scaffolds; templates under the skill don't need
the workaround because their path isn't .github/workflows/.
🐵 - Generated by Copilot
Wire the .github/prompts/ directory into the published artifacts:
- plugin.json: declare 'prompts: .github/prompts/' so the plugin
manifest exposes them alongside agents and skills.
- extension/package.template.json: register all 9 prompt files
(git-ape, agent-{bench,improve,onboard,promote}, skill-{bench,
improve,onboard,promote}) under chatPromptFiles so VS Code picks
them up from the installed extension.
- extension/.vscodeignore: explicitly exclude dev-only .github
subtrees (actionlint, dependabot, aw, copilot, evals, plugins,
references, scripts, templates, workflows). Keeps agents/, skills/,
plugin/, copilot-instructions.md, and now prompts/ in the VSIX
while shedding ~MB of CI tooling that shouldn't ship to users.
🧩 - Generated by Copilot
…t Stacks Align copilot-instructions with the actual workflow templates shipped by the onboarding skill: use 'az stack sub' instead of 'az deployment sub' / 'az group delete' for the full plan-deploy-destroy lifecycle. Why this matters for agents reading the instructions: - The stack is the single unit of lifecycle — create, update, and destroy all operate on it, not on the underlying RGs. - 'deleteAll' on unmanage cleans up every managed resource across every scope (subscription, multiple RGs, sub-scope role/policy assignments) in one call. No orphans, idempotent re-runs. - See Azure#30 for the design rationale. Sample workflow snippet now also passes --action-on-unmanage deleteAll, --deny-settings-mode none, --yes — matching what .github/skills/git-ape-onboarding/templates/workflows/git-ape-deploy.yml generates in target repos. 📘 - Generated by Copilot
scripts/generate-docs.js: teach the workflow doc generator about two
source directories, the existing CI workflows under .github/workflows/
and the new user-facing templates under .github/skills/git-ape-
onboarding/templates/workflows/. Templated workflows get a Docusaurus
:::info admonition explaining they're scaffolded by /git-ape-onboarding
and don't run in the git-ape repo itself. Drops .exampleyml handling
since those stubs are gone.
README.md: update the Workflows table + repo tree to reflect the new
layout. The four git-ape-{plan,deploy,destroy,verify}.exampleyml stubs
no longer exist in .github/workflows/; their canonical sources are
inside the onboarding skill's templates/ directory and scaffolded into
user repos as ready-to-run .yml files. Mention skip-on-collision so
readers know existing workflows are never overwritten.
website/docs/: regenerate every page that the generator touches:
- workflows/{git-ape-plan,deploy,destroy,verify}.md: relocated to the
template source path + new admonition
- workflows/git-ape-drift-lock.md, git-ape-onboarding-template-check.md
(new pages)
- workflows/overview.md: refreshed listing
- agents/git-ape-onboarding.md, skills/git-ape-onboarding.md,
getting-started/onboarding.md: re-synced from current sources
- reference/{plugin-json,marketplace}.md: re-synced to pick up prompts:
registration and chatPromptFiles entries
📚 - Generated by Copilot
8 tasks
…source The auto-generated 'Continuous Drift Remediation' page documents the compiled '.lock.yml' shape. This adds the missing hand-curated page documenting the agentic '.md' source — schedule, severity model, anti-flapping rules, safe-outputs configuration, and how to recompile after editing. Ported from the private repo with two small adaptations: - Workflow-file path updated to the template location under .github/skills/git-ape-onboarding/templates/workflows/git-ape-drift.md (matches the autogen lock-page convention). - Added the ':::info[Scaffolded by /git-ape-onboarding]' admonition for consistency with the autogen lock page; clarifies the file is shipped as a template, not run in the git-ape repo itself. - Added a Related section linking to the lock-page, the azure-drift-detector skill, the deployment guide, and the use-case overview so readers can navigate the full drift story. Marked HAND-CURATED at the top so generate-docs.js maintainers know not to add a generator branch for '.md' workflow sources. 🌊 - Generated by Copilot
Member
Author
|
Superseded by #142, which is the same commits ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
End-to-end overhaul of the
/git-ape-onboardingflow plus the supporting packaging, instructions, eval registration, and regenerated docs. Five themed commits, each independently revertable:chore(models): rename gpt-5-codex to gpt-5.3-codexfeat(onboarding): replace exampleyml stubs with template-driven scaffold.github/workflows/(where the.exampleymlhack lived) into.github/skills/git-ape-onboarding/templates/; ship sync scripts; add eval suite + CI parity check; register the eval in the pilot tierfeat(extension): register prompt files in VSIX and tighten .vscodeignoreplugin.jsonnow declaresprompts:;package.template.jsonregisters all 9 prompts aschatPromptFiles;.vscodeignoresheds dev-only.githubsubtrees from the published VSIXdocs(instructions): switch deploy/destroy guidance to Azure Deployment Stacksaz stack sub create/delete --action-on-unmanage deleteAllinstead ofaz deployment sub+az group delete(see #30)docs(website): regenerate for templated workflows and prompt assets.github/workflows/and the skill templates directory, tags scaffolded workflows with a Docusaurus admonitionKey change:
.exampleymlis goneOld shape (this repo's
.github/workflows/):.exampleymlwas a workaround so GitHub Actions wouldn't auto-load the scaffolds.New shape (
.github/skills/git-ape-onboarding/templates/workflows/):The path is no longer
.github/workflows/so the workaround isn't needed./git-ape-onboardingcopies these into the target repo with skip-on-collision so customized workflows are never overwritten.Eval registration
Adds
git-ape-onboardingtopilottier in .github/evals/manifest.yaml — matches its prior 4-model bench coverage. The eval ships 4 tasks:positive-first-time-setup,positive-multi-env,positive-skip-on-collision,negative-storage-comparison. Closes part of #93.Verification done
actionlintclean on the newgit-ape-onboarding-template-check.ymlnode scripts/generate-docs.jsre-runs with no further driftscripts/carry the executable bitDependency
Depends on #140 (LLM-as-judge →
claude-opus-4.7). This PR is mergeable independently — if it lands first, the newgit-ape-onboardingeval will run with whatever judge is pinned at merge time, then automatically pick up the opus judge once #140 merges. Prefer to merge #140 first to avoid a mixed-judge snapshot.Risk
Medium-low.
.exampleymldeletions are the only destructive change in this repo; their content is preserved verbatim in the templates directory.waza-evalsmatrix dispatch (pilot × 4 models = 16 legs). Quota cost: ~equivalent to prereq-check baseline.