CXF-221: agent skills for post-sync verification, update/rollback, and failure diagnosis - #15
Merged
Merged
Conversation
Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…nd credential guidance - update-and-rollback/SOURCES.md: drop the fabricated RollbackRevision proto RPC citation (the proto exposes no rollback RPC; the REST-only contract lives in the lifecycle doc) — restores decision-5 provenance integrity. - update-and-rollback/SKILL.md: add poll-with-backoff + evidence-unsatisfied guardrail to the ACTIVE poll (mirrors deploy-and-activate); add OWNER bearer-token hygiene anti-pattern. - diagnose-authoring-failure/SKILL.md: bound the evidence polling with backoff and a ~10-poll stop. - verify-connector-output/SKILL.md: handle RUNNING/unknown sync status in the status gate. - skills_bundle.test.ts: lock the rotation-STOP safety negation literal; add test (f) asserting every skill ships a non-empty SOURCES.md and the three new skills name the c1 pin. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…ety prose - verify-connector-output/SKILL.md: the RUNNING-status poll now stops after ~10 polls (STOP and report) instead of polling unbounded; SYNC_STATUS_DISABLED is called out as normal (per the sync_disabled caveat) instead of being caught by the unknown-status catch-all. - skills_bundle.test.ts: lock the new safety prose in SKILL_LITERALS — SYNC_STATUS_RUNNING (verify), evidence is unsatisfied (update-and-rollback), Poll with backoff (diagnose-authoring-failure). Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The f8dcfb1 literal lock for update-and-rollback ('evidence is unsatisfied') was added against a working-tree rewrap that was never committed; the pushed branch still wrapped the phrase across lines and failed test (c) on a clean checkout. Commit the rewrap so the locked literal is present verbatim. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
- verify-connector-output/SKILL.md: SYNC_STATUS_DISABLED is normal only for
the post-activation sync_disabled state; a DISABLED row with
status.lastError set (e.g. a data-anomaly auto-pause) routes through
diagnose-authoring-failure. The backend derives DISABLED only from an
ERROR-classified sync recategorized as ops pause / customer opt-out /
data-anomaly (ConnectorStatusToAPI), so the blanket 'normal' carve-out
would silently certify an anomaly-paused connector as verified.
- skills_bundle.test.ts: lock the poll-termination bounds themselves
('DONE/ERROR after ~10 polls, STOP and report', 'row after ~10 polls,
stop and report') and the data-anomaly distinction, so a future edit
cannot strip the bound clauses while the suite stays green.
Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…emaining safety prose - verify-connector-output/SKILL.md: SYNC_STATUS_DISABLED is an error-classified outcome; the discriminator is now the connector's sync_disabled_category, not status.lastError presence (every real DISABLED row carries a non-empty lastError via ConnectorSyncStatusSetError, so the old gate misrouted benign ops/customer pauses). A data-anomaly auto-pause (SYNC_DISABLED_CATEGORY_DATA_ANOMALY) means the sync was paused after repeated data drops - investigate the counts and the drop reason; a deliberate pause (customer opt-out, ops, or deployment) is normal. - skills_bundle.test.ts: lock SYNC_DISABLED_CATEGORY_DATA_ANOMALY (verify), the ACTIVE-poll bound and the OWNER bearer-token hygiene bullet (update-and-rollback). Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…lated field) - verify-connector-output/SKILL.md: sync_disabled_category is never populated on the app_v1.Connector returned by c1_connector_service_get (mdapp.ConnectorToAPI omits it), so the round-5 discriminator was unreadable and every real DISABLED row fell through to the unknown-status branch. Key the discriminator on sync_disabled_reason instead, which IS populated: the data-anomaly auto-pause stamps the 'Sync paused due to significant drop in sync data' prefix (SyncPausedPrefix), while deliberate pauses use 'system' / 'system-customer-opt-out'. Drop the unreachable 'deployment' category (ConnectorStatusToAPI promotes only OPERATIONS/CUSTOMER_OPT_OUT/DATA_ANOMALY to DISABLED). - verify-connector-output/SOURCES.md: provenance row for the DISABLED semantics from the c1 Go source at the same pin. - skills_bundle.test.ts: lock 'significant drop in sync data' instead of the unreadable category enum. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…llback fail-closed routing
- verify-connector-output/SKILL.md: an empty or unexpected
sync_disabled_reason routes through diagnose-authoring-failure (covers the
ops Mode-B residual state where the reason is cleared while the DISABLED
status remains).
- update-and-rollback/SKILL.md: scope 'Do not redeem the approval token' to
activation ('Do not redeem the activation approval token') so it no longer
reads as contradicting the rollback section's instructed token redemption;
add fail-closed routing for rollback precondition errors.
- skills_bundle.test.ts: update the update-and-rollback literal to the
scoped wording (other skills' literals unchanged).
Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…ol rollback guidance
- update-and-rollback/SKILL.md: scope the no-redeem instruction via prose
('Do not redeem the approval token for activation') in both step 5 and the
Anti-patterns bullet, preserving the plan-locked literal 'Do not redeem
the approval token' verbatim; reword the rollback fail-closed guidance to
the Blocker protocol instead of routing to diagnose-authoring-failure
(whose frontmatter disclaims rollback coverage).
- skills_bundle.test.ts: revert the update-and-rollback literal to the
plan-locked value.
Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
- update-and-rollback/SKILL.md step 7: poll c1_connector_service_get with backoff (every 5-10s) until SYNC_STATUS_DONE; if no DONE after ~10 polls, STOP and report — mirrors the bounded-poll convention used by every other async-wait check in the bundle (ForceSync is async, minutes-to-hours). - skills_bundle.test.ts: lock the new bound literal. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
- update-and-rollback/SKILL.md step 7: after the bounded DONE poll, an SYNC_STATUS_ERROR row routes through diagnose-authoring-failure with status.lastError; an SYNC_STATUS_DISABLED row routes through diagnose-authoring-failure unless sync_disabled_reason indicates a deliberate pause (customer opt-out or ops) — mirrors verify-connector-output's taxonomy so an immediate sync failure is not masked by the poll budget. - skills_bundle.test.ts: lock SYNC_STATUS_ERROR in update-and-rollback. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
…atter carve-out - update-and-rollback/SKILL.md step 7: the poll now exits on any terminal status (DONE, ERROR, or DISABLED) so an immediate sync failure does not burn the ~10-poll budget; bound reworded to 'no terminal status after ~10 polls, STOP and report'. - update-and-rollback/SOURCES.md: provenance row for the ERROR/DISABLED terminal-state semantics from the c1 Go source at the same pin. - diagnose-authoring-failure/SKILL.md: narrow the rollback anti-trigger to 'updating or rolling back a healthy live connector' so a failed-sync route from update-and-rollback step 7 does not collide with the disclaimer. - skills_bundle.test.ts: lock 'no terminal status after ~10 polls, STOP and report' and SYNC_STATUS_DISABLED in update-and-rollback. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
- verify-connector-output/SKILL.md step 1: the RUNNING poll now exits on DONE, ERROR, or DISABLED (a mid-poll data-anomaly auto-pause no longer burns the ~10-poll budget before the DISABLED branch applies) — mirrors update-and-rollback step 7's terminal-state race. - skills_bundle.test.ts: lock 'terminal status after ~10 polls, STOP and report' for verify-connector-output. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
btipling
changed the title
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…ills The seven prior skills are five funnel skills and two pre-1 skills (design-access-model, source-openapi-spec); the intro now says so. Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Contributor
General PR Review: CXF-221: agent skills for post-sync verification, update/rollback, and failure diagnosisBlocking Issues: 0 | Suggestions: 3 | Threads Resolved: 0 Review SummaryScanned the full PR diff for security and correctness: three new skill bodies plus SOURCES.md, the bundle bump to Security IssuesNone found. Correctness IssuesNone found. Suggestions
Prompt for AI agents |
- update-and-rollback: state the rollback actor (agent) and OWNER token provenance (env/secret store, never pasted into chat) - verify-connector-output: name c1_connector_service_force_sync and the poll-to-terminal-status bound in the ID-stability step - diagnose-authoring-failure: clarify that the baton-admin skill's repo-local CLI tooling is replaced by the tenant MCP tools - evals/skills-bundle/README.md: note the three new skills' c1 pin Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
![C1 Squire [DEV]](https://avatars.githubusercontent.com/in/2846581?s=40&u=9dfdc6298bb22a241aaf41a6558bc1563c5c6466&v=4){kind=avatar}
Contributor
Author
|
Addressed all six suggestions:
|
Contributor
Author
|
Addressed all three suggestions (evidence below; no code change needed for any):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Agents authoring connectors in the eval harness could build, test, deploy, and activate a connector, but nothing covered what happens after activation: verifying the synced output, updating or rolling back a live connector, or diagnosing a failed build, draft test, or sync. This PR closes that gap with three new skills —
verify-connector-output,update-and-rollback, anddiagnose-authoring-failure— each with aSOURCES.mdnaming its pinned sources, registered in the eval skill bundle at v0.4.0 with the locked machine-checkable contract extended to cover them.Linear
CXF-221: CXF-70 PR 6: verify / update-rollback / diagnose skills (8, 9, 10)
Linear requirements
This PR meets all requirements of CXF-221.
verify-connector-outputskill: post-sync verification — counts (fixture parity), grant wiring, ID stability across re-sync, UI spot-check; "never invent data to make a demo appear complete"skills/verify-connector-output/SKILL.mdL13-45 (checklist), L47-58 (exit criteria)update-and-rollbackskill: same-catalog rerun; image-digest reuse rule + rotation STOP/escalate limitation; REST-only rollback with OWNER tokenskills/update-and-rollback/SKILL.mdL13-42 (update flow), L44-55 (rotation STOP), L57-78 (rollback)diagnose-authoring-failureskill: symptom → cause → fix router on the docs' common-failures table + draft-test FAIL reading + where logs live; port baton-admin taxonomy, replace all repo toolingskills/diagnose-authoring-failure/SKILL.mdL25-37 (router table, all nine rows), L39-52 (FAIL reading), L54-59 (logs)## Exit criteriain all three SKILL.md bodies; enforced byevals/runner/skills_bundle.test.tstest (c) L186-196evals/skills-bundle/bundle.jsonL2, L11-13evals/runner/skills_bundle.test.tsL18/L201/L250;agent.test.tsL81;scenario.test.tsL103; three scenario JSONsSOURCES.mdfiles;skills/README.mdL16-18; test (f)skills_bundle.test.tsL256-270Release Notes
The connector-authoring skill bundle (
evals/skills-bundle) now ships ten skills at v0.4.0 instead of seven. Agents authoring connectors gain three new skills:verify-connector-output(post-sync verification of counts, grant wiring, and ID stability),update-and-rollback(same-catalog updates and REST-only rollback of live connectors), anddiagnose-authoring-failure(symptom-to-cause-to-fix routing for build, draft-test, activation, and sync failures).What changed
skills/verify-connector-output/SKILL.md(new, 74 lines)Where: whole file L1-74; checklist L13-45, exit criteria L47-58.
What changed
SYNC_STATUS_DONE;SYNC_STATUS_DISABLEDsemantics viasync_disabled_reason), count parity against the live tenant API, grant wiring (every principal → emitted resource, every entitlement ID → emitted entitlement), ID stability across a re-sync, UI spot-check at/admin/connector/<catalog_id>/<app_id>/<connector_id>.Why
probe-contracts.mdassertion inventory.skills/verify-connector-output/SOURCES.md(new, 13 lines)Where: whole file L1-13.
What changed
2e5f53eb…, marketplaceprobe-contracts.md@0cc5ac2a…, c1 Go source) and what each contributes.Why
skills/update-and-rollback/SKILL.md(new, 115 lines)Where: whole file L1-115; update flow L13-42, rotation STOP L44-55, rollback L57-78.
What changed
list_revision_summariesuntil ACTIVE and recordactivation_epoch, force sync, poll to terminal status.serve image does not match the revision-pinned runtime image; do not clear runtime fields, call the provisioner directly, or mutate the deployment or AWS resources; record tenant/catalog/app/connector/target-revision IDs; escalate).POST /api/v1/connector-authoring/rollbackswithcatalog_id,target_revision_id,instance_app_id,instance_connector_id,approval_token_id; strictly-greater activation epoch pointer move.Why
authoring.proto, and the lifecycle doc's "Updating and rolling back a live connector" section.skills/update-and-rollback/SOURCES.md(new, 12 lines)Where: whole file L1-12.
What changed
authoring.proto(MintApprovalToken,ListRevisionSummaries,RevisionStatus), lifecycle doc (image-digest rule, rotation STOP, rollback body), c1 Go source (SYNC_STATUS_ERROR/SYNC_STATUS_DISABLEDsemantics).Why
skills/diagnose-authoring-failure/SKILL.md(new, 90 lines)Where: whole file L1-90; router table L25-37, FAIL reading L39-52, logs L54-59.
What changed
is_secret, credential re-entry, missingtype, unregistered transport,ticketing.enabled, activation evidence unsatisfied,Invalid token provided).ConnectionOK,HostCallOK, no read error and no write attempt, config version handle match, runtime image digest match.c1_connector_service_get→status.status,status.lastError).Why
diagnose-connector-failuretaxonomy approach and replaces all repo tooling.skills/diagnose-authoring-failure/SOURCES.md(new, 13 lines)Where: whole file L1-13.
What changed
authoring.proto(GetTestRunEvidence), lifecycle doc (common-failures table, FAIL reading, logs), baton-admindiagnose-connector-failure@6fe6886f….Why
evals/skills-bundle/bundle.json(modified)Where: L2, L11-13.
What changed
version0.3.0 → 0.4.0; three new entries (verify-connector-output,update-and-rollback,diagnose-authoring-failureat 0.1.0); the existing seven entries byte-preserved.Why
bundle.json's.evals/runner/skills_bundle.test.ts(modified)Where: L17-18, L59, L75-184, L201, L250, L256-270.
What changed
SKILLSlist 7 → 10,VERSION0.4.0, count assertion 7 → 10, three newSKILL_LITERALSentries (the locked literal minimums per skill), 0.3.0 → 0.4.0 in tests (d) and (e), and a new test (f) asserting every skill ships a non-emptySOURCES.mdnaming the c1 pin.Why
evals/runner/agent.test.ts(modified)Where: L81.
What changed
skillBundlefixture version 0.3.0 → 0.4.0.Why
evals/runner/scenario.test.ts(modified)Where: L103.
What changed
Why
evals/scenarios/tier1-directory-full.json(modified)Where: L30.
What changed
skillBundle.version0.3.0 → 0.4.0.Why
bundle.json's.evals/scenarios/pre1-directory-proceed.json(modified)Where: L32.
What changed
skillBundle.version0.3.0 → 0.4.0.Why
evals/scenarios/pre1-noiam-park.json(modified)Where: L30.
What changed
skillBundle.version0.3.0 → 0.4.0.Why
evals/skills-bundle/README.md(modified)Where: L5-20, L62.
What changed
## v0.4.0 — the ten skillssection describing the three new skills; the mount-contract pin updated 0.2.0 → 0.4.0.Why
skills/README.md(modified)Where: L1-10, L16-18.
What changed
Why
evals/README.md(modified)Where: L18, L245.
What changed
Why
Excluded
No generated files. The eval runner test files are the deliverable's machine-checkable contract (plan Phase C), not incidental scaffolding — covered above. The READMEs are the provenance contract (plan Phase B) — covered above.