Skip to content

feat(onboarding): template-driven scaffold + register prompts/eval in matrix#141

Closed
arnaudlh wants to merge 6 commits into
Azure:mainfrom
arnaudlh:feat/onboarding-overhaul
Closed

feat(onboarding): template-driven scaffold + register prompts/eval in matrix#141
arnaudlh wants to merge 6 commits into
Azure:mainfrom
arnaudlh:feat/onboarding-overhaul

Conversation

@arnaudlh
Copy link
Copy Markdown
Member

Summary

End-to-end overhaul of the /git-ape-onboarding flow plus the supporting packaging, instructions, eval registration, and regenerated docs. Five themed commits, each independently revertable:

# Commit Why
1 chore(models): rename gpt-5-codex to gpt-5.3-codex waza catalog renamed; aligns manifest tiers + bench prompt defaults
2 feat(onboarding): replace exampleyml stubs with template-driven scaffold Move workflow scaffolds out of .github/workflows/ (where the .exampleyml hack lived) into .github/skills/git-ape-onboarding/templates/; ship sync scripts; add eval suite + CI parity check; register the eval in the pilot tier
3 feat(extension): register prompt files in VSIX and tighten .vscodeignore plugin.json now declares prompts:; package.template.json registers all 9 prompts as chatPromptFiles; .vscodeignore sheds dev-only .github subtrees from the published VSIX
4 docs(instructions): switch deploy/destroy guidance to Azure Deployment Stacks Copilot-instructions caught up with the actual templates: az stack sub create/delete --action-on-unmanage deleteAll instead of az deployment sub + az group delete (see #30)
5 docs(website): regenerate for templated workflows and prompt assets Generator now scans both .github/workflows/ and the skill templates directory, tags scaffolded workflows with a Docusaurus admonition

Key change: .exampleyml is gone

Old shape (this repo's .github/workflows/):

git-ape-plan.exampleyml
git-ape-deploy.exampleyml
git-ape-destroy.exampleyml
git-ape-verify.exampleyml

.exampleyml was a workaround so GitHub Actions wouldn't auto-load the scaffolds.

New shape (.github/skills/git-ape-onboarding/templates/workflows/):

git-ape-plan.yml
git-ape-deploy.yml
git-ape-destroy.yml
git-ape-verify.yml
git-ape-drift.md          # agentic workflow source
git-ape-drift.lock.yml    # compiled lockfile

The path is no longer .github/workflows/ so the workaround isn't needed. /git-ape-onboarding copies these into the target repo with skip-on-collision so customized workflows are never overwritten.

Eval registration

Adds git-ape-onboarding to pilot tier in .github/evals/manifest.yaml — matches its prior 4-model bench coverage. The eval ships 4 tasks: positive-first-time-setup, positive-multi-env, positive-skip-on-collision, negative-storage-comparison. Closes part of #93.

Verification done

  • actionlint clean on the new git-ape-onboarding-template-check.yml
  • node scripts/generate-docs.js re-runs with no further drift
  • Eval YAML files parse
  • All shell scripts under scripts/ carry the executable bit

Dependency

Depends on #140 (LLM-as-judge → claude-opus-4.7). This PR is mergeable independently — if it lands first, the new git-ape-onboarding eval will run with whatever judge is pinned at merge time, then automatically pick up the opus judge once #140 merges. Prefer to merge #140 first to avoid a mixed-judge snapshot.

Risk

Medium-low.

  • Workflow templates are new files in a non-loaded path, no production CI impact.
  • The 4 .exampleyml deletions are the only destructive change in this repo; their content is preserved verbatim in the templates directory.
  • Eval registration adds 4 new task runs to the next waza-evals matrix dispatch (pilot × 4 models = 16 legs). Quota cost: ~equivalent to prereq-check baseline.

arnaudlh added 5 commits May 29, 2026 16:22
The waza model catalog now ships gpt-5-codex under its versioned ID
gpt-5.3-codex. Align manifest tiers and bench-prompt argument hints
so dispatched runs resolve to a valid model.

- .github/evals/manifest.yaml: pilot + expanded tier model lists
- .github/prompts/agent-bench.prompt.md: default models in argument-hint + body
- .github/prompts/skill-bench.prompt.md: default models in argument-hint + body

🔖 - Generated by Copilot
Rewrite git-ape-onboarding as a skill-driven CLI playbook backed by a
sync-able template bundle. The previous .exampleyml workflows lived in
this repo's .github/workflows/ and were copy-pasted by users; they're
now first-class templates under the skill and pushed into target repos
by scripts/sync-templates.{sh,ps1}.

What ships:
- .github/agents/git-ape-onboarding.agent.md: rewritten flow + tools
- .github/skills/git-ape-onboarding/SKILL.md: new playbook structure
- .github/skills/git-ape-onboarding/scripts/: bash + pwsh helpers
    - scaffold-repo.{sh,ps1}: bootstrap target repo
    - sync-templates.{sh,ps1}: drop-in workflow + instructions update
- .github/skills/git-ape-onboarding/templates/: canonical target-repo
  artifacts (copilot-instructions.md, workflows/git-ape-{plan,deploy,
  destroy,verify,drift}.yml + drift agentic workflow + drift lockfile)
- .github/evals/git-ape-onboarding/: positive + negative tasks for
  first-time-setup, multi-env, skip-on-collision, and storage refusal
- .github/workflows/git-ape-onboarding-template-check.yml: CI check
  that the shipped templates pass actionlint and round-trip cleanly
- .github/evals/manifest.yaml: register git-ape-onboarding in pilot
  tier (matches its prior 4-model bench coverage)

Removed:
- .github/workflows/git-ape-{deploy,destroy,plan,verify}.exampleyml:
  retired — their content is now in skills/.../templates/workflows/

The .exampleyml extension was a workaround to keep GitHub Actions from
auto-loading workflow scaffolds; templates under the skill don't need
the workaround because their path isn't .github/workflows/.

🐵 - Generated by Copilot
Wire the .github/prompts/ directory into the published artifacts:

- plugin.json: declare 'prompts: .github/prompts/' so the plugin
  manifest exposes them alongside agents and skills.
- extension/package.template.json: register all 9 prompt files
  (git-ape, agent-{bench,improve,onboard,promote}, skill-{bench,
  improve,onboard,promote}) under chatPromptFiles so VS Code picks
  them up from the installed extension.
- extension/.vscodeignore: explicitly exclude dev-only .github
  subtrees (actionlint, dependabot, aw, copilot, evals, plugins,
  references, scripts, templates, workflows). Keeps agents/, skills/,
  plugin/, copilot-instructions.md, and now prompts/ in the VSIX
  while shedding ~MB of CI tooling that shouldn't ship to users.

🧩 - Generated by Copilot
…t Stacks

Align copilot-instructions with the actual workflow templates shipped
by the onboarding skill: use 'az stack sub' instead of 'az deployment
sub' / 'az group delete' for the full plan-deploy-destroy lifecycle.

Why this matters for agents reading the instructions:
- The stack is the single unit of lifecycle — create, update, and
  destroy all operate on it, not on the underlying RGs.
- 'deleteAll' on unmanage cleans up every managed resource across
  every scope (subscription, multiple RGs, sub-scope role/policy
  assignments) in one call. No orphans, idempotent re-runs.
- See Azure#30 for the design rationale.

Sample workflow snippet now also passes --action-on-unmanage deleteAll,
--deny-settings-mode none, --yes — matching what
.github/skills/git-ape-onboarding/templates/workflows/git-ape-deploy.yml
generates in target repos.

📘 - Generated by Copilot
scripts/generate-docs.js: teach the workflow doc generator about two
source directories, the existing CI workflows under .github/workflows/
and the new user-facing templates under .github/skills/git-ape-
onboarding/templates/workflows/. Templated workflows get a Docusaurus
:::info admonition explaining they're scaffolded by /git-ape-onboarding
and don't run in the git-ape repo itself. Drops .exampleyml handling
since those stubs are gone.

README.md: update the Workflows table + repo tree to reflect the new
layout. The four git-ape-{plan,deploy,destroy,verify}.exampleyml stubs
no longer exist in .github/workflows/; their canonical sources are
inside the onboarding skill's templates/ directory and scaffolded into
user repos as ready-to-run .yml files. Mention skip-on-collision so
readers know existing workflows are never overwritten.

website/docs/: regenerate every page that the generator touches:
- workflows/{git-ape-plan,deploy,destroy,verify}.md: relocated to the
  template source path + new admonition
- workflows/git-ape-drift-lock.md, git-ape-onboarding-template-check.md
  (new pages)
- workflows/overview.md: refreshed listing
- agents/git-ape-onboarding.md, skills/git-ape-onboarding.md,
  getting-started/onboarding.md: re-synced from current sources
- reference/{plugin-json,marketplace}.md: re-synced to pick up prompts:
  registration and chatPromptFiles entries

📚 - Generated by Copilot
@arnaudlh arnaudlh linked an issue May 29, 2026 that may be closed by this pull request
8 tasks
@arnaudlh arnaudlh added the AI-evals All things related to agent and skills evaluation. label May 29, 2026
@arnaudlh arnaudlh requested a review from sendtoshailesh May 29, 2026 08:36
…source

The auto-generated 'Continuous Drift Remediation' page documents the
compiled '.lock.yml' shape. This adds the missing hand-curated page
documenting the agentic '.md' source — schedule, severity model,
anti-flapping rules, safe-outputs configuration, and how to recompile
after editing.

Ported from the private repo with two small adaptations:
- Workflow-file path updated to the template location under
  .github/skills/git-ape-onboarding/templates/workflows/git-ape-drift.md
  (matches the autogen lock-page convention).
- Added the ':::info[Scaffolded by /git-ape-onboarding]' admonition for
  consistency with the autogen lock page; clarifies the file is shipped
  as a template, not run in the git-ape repo itself.
- Added a Related section linking to the lock-page, the
  azure-drift-detector skill, the deployment guide, and the use-case
  overview so readers can navigate the full drift story.

Marked HAND-CURATED at the top so generate-docs.js maintainers know
not to add a generator branch for '.md' workflow sources.

🌊 - Generated by Copilot
@arnaudlh
Copy link
Copy Markdown
Member Author

Superseded by #142, which is the same commits (8d5b9cf5..e66e2ecc) pushed to a branch on Azure/git-ape itself so the waza eval matrix can read COPILOT_GITHUB_TOKEN. Fork PRs were skipping all eval legs at preflight. Closing this one to keep review in a single thread.

@arnaudlh arnaudlh closed this May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-evals All things related to agent and skills evaluation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Author eval suite for skill git-ape-onboarding

1 participant