Skip to content

Add ADO Services support to respond-to-pr-comments + new human-voice-fidelity protocol#254

Open
abeltrano wants to merge 12 commits intomainfrom
add-human-voice-fidelity-protocol
Open

Add ADO Services support to respond-to-pr-comments + new human-voice-fidelity protocol#254
abeltrano wants to merge 12 commits intomainfrom
add-human-voice-fidelity-protocol

Conversation

@abeltrano
Copy link
Copy Markdown
Collaborator

@abeltrano abeltrano commented May 4, 2026

Note

This PR supersedes #253. It bundles the Azure DevOps Services
support for respond-to-pr-comments with the new
human-voice-fidelity guardrail protocol. Both sets of changes are
in this branch; #253 is now closed.

This PR delivers two related improvements to how PromptKit handles
externally visible text on the user's behalf:

  1. respond-to-pr-comments now supports Azure DevOps Services
    alongside GitHub. The workflow shape is shared and only the API
    recipes branch per platform; users do not need to pass a new
    parameter or pick a platform.
  2. A new opt-in guardrail protocol — human-voice-fidelity
    preserves the user's communication style when an agent drafts
    externally visible text on their behalf (PR replies, issue
    comments, review comments, emails, chat messages). It is wired
    into the two templates that post text under the user's identity
    today: respond-to-pr-comments and review-pull-request.

Part 1 — ADO Services support for respond-to-pr-comments

What's new

  • Auto-detect platform from the PR URL (github.com vs
    dev.azure.com / *.visualstudio.com), with fallback to
    git remote -v (handles SSH remotes including
    git@ssh.dev.azure.com:v3/... and
    {org}@vs-ssh.visualstudio.com:v3/...). Prompts on ambiguity —
    does not guess.
  • ADO auth via az login + az rest --resource <ADO GUID> on
    every call. No Personal Access Token path.
  • Native status vocabulary preserved — no cross-platform
    normalization. ADO uses fixed (not resolved) per the official
    CommentThreadStatus REST enum.
  • ADO reply payload uses content + parentCommentId +
    commentType: "text". parentCommentId is always set, including
    for PR-wide threads (reply to the latest text comment).
  • System threads are filtered (commentType: "system" or system
    CodeReviewThreadType properties); threads with no text comments
    are flagged for the user, not auto-skipped.
  • Conservative outdated detection — prefer the ADO iterations +
    items APIs; fall back to the local working tree only when HEAD
    matches the iteration's source-branch tip; otherwise mark
    "unknown / not verified" and ask. PR-wide threads (null
    threadContext) skip outdated detection.
  • GitHub recipe paginates both outer reviewThreads and inner
    comments via follow-up cursored queries.
  • Out of scope, fail fast: ADO Server / on-prem / TFS / custom
    hostnames stop with a clear message rather than attempting
    unsupported API calls.

Files changed (Part 1)

  • templates/respond-to-pr-comments.md — added Phase 1 platform
    detection and per-platform branches in Phase 2 (gather) and Phase
    5 (apply); ADO edge cases in Phase 6; expanded Quality Checklist.
  • .github/skills/respond-to-pr-comments/SKILL.md — mirrors the
    template with compact per-platform recipes side-by-side.
  • formats/pr-comment-responses.md — generalized status tables and
    the Action Summary to use platform-native vocabulary; tables use
    the exact API enum literals (active, fixed, wontFix,
    byDesign, etc.) for consistency with surrounding text.
  • manifest.yaml — updated entry description to mention ADO
    Services.

Part 2 — human-voice-fidelity guardrail protocol

The protocol is scoped narrowly to user-authored prose. Internal
analysis, code, command output, format scaffolding, and quoted
reviewer text are all exempt by design.

Protocol features

  • Pluggable voice sources in priority order: explicit samples in
    the current session, prior repo PRs/MRs by the user (with
    per-platform recipes for GitHub, Azure DevOps Services, GitLab,
    Bitbucket Cloud, Gitea/Forgejo, plus a fallback for unspecified
    SCMs), prior agent session history (Copilot CLI, Claude Code,
    Cursor, etc.), organization-specific tools (e.g. Microsoft 365
    Copilot / WorkIQ MCP), explicit style notes (STYLE.md,
    CLAUDE.md, AGENTS.md, .cursorrules, etc.).
  • Calibrated style extraction — sentence length distribution,
    hedging frequency, technical density, openers/closers,
    characteristic phrases, punctuation habits.
  • Per-output self-check with conditional rules: AI-tell patterns
    (em-dash, formulaic phrases, etc.) are flagged unless they appear
    in the user's own samples, plus soft rules covering length budget,
    greeting/sign-off, and opener.
  • Neutral collaborative default + explicit disclosure when no
    samples are available. Never claims voice match without evidence.
  • Output annotation — a brief Voice Calibration note (internal,
    not posted externally).

Integration

  • protocols/guardrails/human-voice-fidelity.md (new).
    applicable_to: [respond-to-pr-comments, review-pull-request].
  • manifest.yaml — register protocol under protocols.guardrails;
    add to both consuming templates' protocols lists.
  • templates/respond-to-pr-comments.md — add to frontmatter;
    reference the protocol when drafting reviewer replies, with
    explicit scope note that exempts surrounding analysis, code, and
    quoted text. Points to the protocol's Phase 4 self-check rather
    than restating rules inline (avoids drift).
  • templates/review-pull-request.md — add to frontmatter; reference
    the protocol in Phase 5 action-mode where inline review comment
    bodies and the overall review summary body are drafted before
    being POSTed to the GitHub Reviews API.
  • .github/skills/respond-to-pr-comments/SKILL.md — mirrors the
    template's draft-reply guidance.

Other candidate consumers (not in this PR)

A scan of the manifest identified additional templates where this
protocol may be appropriate as follow-up work:

  • generate-commit-message — commit messages are
    git-history-attributed under the user's identity. The conditional
    self-check catches common AI commit-message tells, though
    calibrated voice extraction has limited surface area given the
    heavy structural conventions (Conventional Commits,
    Problem/Solution). Worth applying in a follow-up PR after we see
    how the protocol performs in real PR-review use.

Templates intentionally not considered candidates:

  • Internal-artifact templates (triage-issues,
    triage-pull-requests, author-session-handoff, all
    audit/review/investigate reports) — output goes to the user, not
    posted externally on their behalf.
  • Document-authoring templates (author-rfc,
    author-architecture-spec, author-presentation,
    author-design-doc, etc.) — institutional voice with strict
    structural conventions; calibrated personal voice is not the right
    model.
  • Code/spec/test/hardware generation templates — produce code,
    schematics, specs, plans, not user-voice prose.

Out of scope (Part 2)

Deferred to potential follow-up PRs (intentionally not included
here):

  • Delegation matrix for action-mode confirmations
  • Work-item proposal phase
  • Format schema expansion (disposition / target-state / work-item
    breakdowns / Voice Calibration section)
  • SKILL rewrite for compactness

Validation

  • python tests/validate-manifest.py passes (manifest ↔ template
    protocol sync check).
  • Protocol's applicable_to and both templates' protocols
    frontmatter are in sync with the manifest entries.
  • Two rubber-duck critique passes were applied to the ADO recipes;
    ADO REST shape verified against MS Learn docs
    (CommentThreadStatus, threads/comments endpoints, payload field
    names).
  • Three rounds of copilot-pull-request-reviewer feedback on this
    branch addressed across commits 8e4ff0d, 7650489, and
    8359da0.

abeltrano and others added 4 commits May 4, 2026 12:12
Extends the respond-to-pr-comments template, skill, and format to

handle Azure DevOps Services PRs alongside GitHub. Platform is

auto-detected from the PR URL (with git-remote fallback); the

workflow shape is shared and only API recipes branch per platform.

- Auto-detect platform from PR URL or git remote (handles SSH and

  legacy visualstudio.com hosts); prompt on ambiguity, do not guess.

- ADO auth uses 'az login' + 'az rest --resource <ADO GUID>' on

  every call; no Personal Access Token path.

- Preserve each platform's native status vocabulary in output (no

  cross-platform normalization). ADO uses 'fixed' (not 'resolved')

  per the CommentThreadStatus REST enum.

- ADO reply payload uses content + parentCommentId + commentType

  ('text'); always set parentCommentId, including for PR-wide threads.

- Filter ADO system threads (commentType 'system' or system

  CodeReviewThreadType properties); flag, do not auto-skip, threads

  with no text comments.

- Conservative outdated detection: prefer ADO iteration/items API,

  fall back to local working tree only when HEAD matches the

  iteration's source-branch tip; otherwise mark unverified.

- GitHub recipe paginates both reviewThreads and inner comments via

  follow-up cursored queries.

- ADO Server / on-prem / TFS / custom hostnames are out of scope —

  stop with a clear message.

- Update format file with per-platform status tables and add

  byDesign to the closed-state action summary.

- Update manifest description to mention ADO Services.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ADO reply POST switched to temp-file pattern (--body @reply.json)

  to handle apostrophes/newlines/backslashes in real reply text;

  mirrors the GitHub recipe.

- pr_reference param doc clarified: URL auto-detect with git-remote

  fallback and ambiguity prompt (covers #42 / !123 inputs).

- All shell command fences in SKILL labeled bash for clarity.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rotocol

Adds a new opt-in guardrail protocol that preserves the user's

communication style when an agent drafts externally visible text on

the user's behalf. Scoped narrowly to user-authored prose; analysis,

code, command output, and quoted reviewer text are exempt.

Protocol features:

- Pluggable voice sources (session samples, prior repo PRs, session

  history, org tools, explicit style notes), in priority order.

- Calibrated style extraction (sentence length, hedging, technical

  density, openers/closers, characteristic phrases, punctuation).

- Per-output self-check that bans em-dashes and a list of AI-tell

  phrases unless they appear in the user's own samples.

- Neutral collaborative default + explicit disclosure when no

  voice samples are available; never claims voice match without

  evidence.

- Output annotation requiring a Voice Calibration note.

Integration:

- protocols/guardrails/human-voice-fidelity.md (new, 163 lines)

- manifest.yaml: register protocol under guardrails.

- respond-to-pr-comments template: add protocol to frontmatter and

  manifest protocols list; reference the protocol when drafting

  reviewer replies.

Out of scope (deferred): delegation matrix, work-item proposal

phase, expanded format schema, and SKILL rewrite from the prior

exploration branch — those will land in separate PRs if pursued.

Validation: python tests/validate-manifest.py passes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds human-voice-fidelity to the review-pull-request template, the

second template that drafts text posted externally under the user's

identity. Action mode of review-pull-request POSTs inline review

comments and an overall review summary to GitHub via the Reviews

API; both bodies are user-voice prose where AI tells (em-dashes,

AI-tell phrases) would betray non-human authorship.

Changes:

- protocols/guardrails/human-voice-fidelity.md: add review-pull-request

  to applicable_to.

- manifest.yaml: add human-voice-fidelity to review-pull-request

  protocols list.

- templates/review-pull-request.md: add to frontmatter; reference the

  protocol in Phase 5 action-mode step where comment bodies are

  drafted, with the same scope note (drafted prose only; code/paths

  /quoted text exempt).

Validation: python tests/validate-manifest.py passes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 4, 2026 20:26
@abeltrano abeltrano changed the title feat(human-voice-fidelity): add opt-in voice-preservation guardrail protocol Add human-voice-fidelity guardrail protocol May 4, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a new opt-in human-voice-fidelity guardrail protocol and wires it into the templates that draft externally posted PR text under the user’s identity.

Changes:

  • Introduces protocols/guardrails/human-voice-fidelity.md and registers it in manifest.yaml.
  • Applies the protocol to externally posted prose in review-pull-request and respond-to-pr-comments.
  • Expands respond-to-pr-comments (and related format/skill docs) to support Azure DevOps Services alongside GitHub, including native status vocabulary guidance.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
templates/review-pull-request.md Adds human-voice-fidelity to frontmatter and instructs applying it to posted review bodies.
templates/respond-to-pr-comments.md Adds the protocol and significantly expands workflow to support GitHub + Azure DevOps Services.
protocols/guardrails/human-voice-fidelity.md New guardrail protocol defining voice sourcing, calibration, and per-output self-check rules.
manifest.yaml Registers the new protocol and adds it to the two consuming templates.
formats/pr-comment-responses.md Updates output format to use platform-native status vocab (GitHub vs ADO).
.github/skills/respond-to-pr-comments/SKILL.md Updates skill guidance for GitHub + ADO support and native status vocabulary.

Comment thread protocols/guardrails/human-voice-fidelity.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Replace the GitHub-only gh recipe in Phase 1 item 2 with a per-platform list (GitHub, Azure DevOps Services, GitLab, Bitbucket Cloud, Gitea/Forgejo) plus a fallback bullet for unspecified SCMs. The underlying intent is unchanged: sample 5-20 recent self-authored comment bodies, preferring inline review-comment bodies over commit messages.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nostic

Item 3 was Copilot-CLI-specific; reframe as 'prior agent session history' with examples for GitHub Copilot CLI (session_store_sql), Claude Code (~/.claude/projects JSONL transcripts), and Cursor/Windsurf/IDE-embedded agents, plus a fallback for other agents.

Item 5 was copilot-instructions.md-only; expand the example list to also include CLAUDE.md, AGENTS.md, .cursorrules, and .windsurfrules.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 4, 2026 20:40
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Comment thread protocols/guardrails/human-voice-fidelity.md Outdated
Comment thread protocols/guardrails/human-voice-fidelity.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread protocols/guardrails/human-voice-fidelity.md
Comment thread templates/respond-to-pr-comments.md Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 10 comments.

Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread .github/skills/respond-to-pr-comments/SKILL.md Outdated
Comment thread .github/skills/respond-to-pr-comments/SKILL.md Outdated
Comment thread protocols/guardrails/human-voice-fidelity.md
Comment thread protocols/guardrails/human-voice-fidelity.md Outdated
Comment thread protocols/guardrails/human-voice-fidelity.md
Comment thread protocols/guardrails/human-voice-fidelity.md
abeltrano and others added 2 commits May 4, 2026 15:13
Voice-fidelity protocol (4 threads):

- Em-dash rule now conditional on user's own samples (T1, T5)

- Bullet-list rule moved from hard rules to soft rules (T6)

- Add consent and confidentiality requirements before sampling

  agent transcripts or org-tool history (T9)

respond-to-pr-comments template/skill (6 threads):

- Soften ADO threads pagination claim and add defensive

  continuationToken handling (T2)

- URL-encode {project} and {repoName} in all ADO az rest example

  URIs via {projectEnc}/{repoNameEnc} placeholders (T3, T4, T10)

- Replace GitLab-style !123 PR id notation with bare 123 or

  ado:123 prefix (T7)

- Rename 'all pending' selector to 'all open' to avoid collision

  with ADO's distinct 'pending' status (T8)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ADO status enum casing (5 threads):

- Replace 'lowercase API enum values' wording in template, skill,

  PATCH instructions, and checklist with 'exact case-sensitive enum

  values' noting wontFix and byDesign are camelCase

Voice fidelity (2 threads):

- Tighten source 4 (org communication tools) to require explicit,

  per-session opt-in; default behavior is to skip

- Tighten Consent block: disclose what, approximate volume, and

  that consent does not carry between sessions

- Clarify Voice Calibration note placement: reported in agent

  chat output / action summary, NOT inserted into the produced

  format artifact (prevents format drift in pr-comment-responses

  and investigation-report)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 4, 2026 21:18
These were accidentally included via git add -A in 7650489. They are

not part of the human-voice-fidelity protocol work.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Comment thread templates/review-pull-request.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread manifest.yaml Outdated
Comment thread formats/pr-comment-responses.md Outdated
Comment thread .github/skills/respond-to-pr-comments/SKILL.md Outdated
- Soften voice-fidelity self-check paraphrases in template, manifest,
  and review-pull-request to reflect the protocol's conditional rules
  (avoid restating the rules to prevent future drift)
- Switch ADO/GitHub status tables in pr-comment-responses format to
  API enum literals for consistency with surrounding text
- Add human-voice-fidelity reference to SKILL.md draft-reply step

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@abeltrano abeltrano changed the title Add human-voice-fidelity guardrail protocol Add ADO Services support to respond-to-pr-comments + new human-voice-fidelity protocol May 4, 2026
…eply text

Serialization to JSON may preserve shell-specific character escape
sequences literally rather than the intended Unicode character. Add
guidance to use literal UTF-8 characters directly in reply text.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 4, 2026 23:18
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread templates/respond-to-pr-comments.md Outdated
Comment thread formats/pr-comment-responses.md Outdated
Comment thread formats/pr-comment-responses.md Outdated
Comment thread .github/skills/respond-to-pr-comments/SKILL.md
- Phase 1: make `ado:` prefix an explicit platform override that
  bypasses remote inspection; carry only the numeric `prId`
- Phase 2 GitHub: reference concrete GraphQL fields
  (`isResolved`/`isOutdated`) instead of a non-existent `state` field
- Phase 2 ADO: reuse Phase 1 coordinates instead of re-parsing the URL
- Format: reframe GitHub status table as derived workflow labels
  (`open`/`outdated`/`resolved`); cascade `pending`->`open` in skill
- Format: standardize per-thread placeholder examples to lowercase
  code literals matching the Thread Summary tables
- SKILL.md: mirror the `ado:` prefix handling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants