Rectify: Sub-Skill Refusal Handling and Diagram Styling Contract Enforcement

[Trecek]

Summary

When open-research-pr is invoked headlessly, the exp-lens sub-skills it depends on remain gated (disable-model-invocation: true) because activate_tier2() un-gates exactly one skill per session by design. The SKILL.md provides no instruction for when the Skill tool refuses a sub-skill invocation, so the model improvises freehand — inventing non-canonical class names and never applying them to nodes. All nodes render gray.

The direct bugs are: (1) no refusal handler in open-research-pr Step 4, and (2) no canonical palette embedded as a reference fallback. The arch-lens equivalent skill open-pr has the identical gap in Step 5. The root architectural weakness is that all contract tests use vocabulary-only assertions — checking if a word appears, not whether a mechanism exists.

Part A adds the two focused failing tests for open-research-pr and open-pr, then fixes both SKILL.md files. Part B (separate task) adds the cross-skill parametrized ratchet tests that enforce the same contract across all sub-skill-calling skills in the codebase.

Architecture Impact

Error/Resilience Diagram

%%{init: {'flowchart': {'nodeSpacing': 40, 'rankSpacing': 50, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef terminal fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;

    subgraph Callers ["● MODIFIED SKILL CALLERS"]
        direction LR
        ORP["● open-research-pr/SKILL.md<br/>━━━━━━━━━━<br/>Step 4: exp-lens invocation<br/>+refusal handler added"]
        OP["● open-pr/SKILL.md<br/>━━━━━━━━━━<br/>Step 5: arch-lens invocation<br/>+refusal handler added"]
    end

    subgraph LensGate ["LENS INVOCATION GATE"]
        SKILL_CALL["Skill tool call<br/>━━━━━━━━━━<br/>/autoskillit:{lens-slug}"]
        GATE{"Error contains<br/>'disable-model-invocation'<br/>or 'cannot be used'?"}
    end

    subgraph RefusalPath ["● REFUSAL HANDLER (added to both skills)"]
        direction TB
        DISCARD["Discard lens silently<br/>━━━━━━━━━━<br/>do NOT write freehand<br/>Continue to next lens"]
        ALL_REFUSED{"All lens invocations<br/>refused?"}
        DIAG["● lens_unavailable_{ts}.txt<br/>━━━━━━━━━━<br/>Diagnostic artifact<br/>open-research-pr only"]
        EMPTY["validated_diagrams = []<br/>━━━━━━━━━━<br/>Diagram section omitted<br/>from PR body"]
    end

    subgraph HappyPath ["UNAFFECTED SUCCESS PATH"]
        direction TB
        EXECUTE["Lens executes<br/>━━━━━━━━━━<br/>Canonical palette applied"]
        VALIDATE["Marker validation<br/>━━━━━━━━━━<br/>★ / ● symbols checked"]
        VALIDATED["validated_diagrams<br/>━━━━━━━━━━<br/>Styled mermaid blocks added"]
    end

    subgraph ContractRatchet ["★ NEW CI CONTRACT ENFORCEMENT"]
        direction TB
        REFUSAL_RATCHET["★ test_sub_skill_refusal_contracts.py<br/>━━━━━━━━━━<br/>Auto-discovers ALL sub-skill callers<br/>Enforces refusal handler at CI time"]
        PALETTE_RATCHET["★ test_mermaid_palette_contracts.py<br/>━━━━━━━━━━<br/>Auto-discovers diagram generators<br/>Enforces palette or mermaid-load"]
        ORP_TESTS["● test_open_research_pr_contracts.py<br/>━━━━━━━━━━<br/>+test_handles_skill_tool_refusal_for_exp_lens<br/>+test_embeds_canonical_classdef_palette"]
        OP_TESTS["● test_open_pr_contracts.py<br/>━━━━━━━━━━<br/>+test_handles_skill_tool_refusal_for_arch_lens"]
    end

    T_PR_STYLED([PR: styled diagram section])
    T_PR_CLEAN([PR: section omitted — clean])
    T_CI_FAIL([CI FAIL: contract violation])
    T_CI_PASS([CI PASS: contracts satisfied])

    ORP --> SKILL_CALL
    OP --> SKILL_CALL
    SKILL_CALL --> GATE
    GATE -->|"no — ungated"| EXECUTE
    GATE -->|"yes — gated/refused"| DISCARD
    DISCARD --> ALL_REFUSED
    ALL_REFUSED -->|"no — more lenses"| SKILL_CALL
    ALL_REFUSED -->|"yes — all refused"| DIAG
    DIAG --> EMPTY
    EMPTY --> T_PR_CLEAN
    EXECUTE --> VALIDATE
    VALIDATE -->|"contains ★ or ●"| VALIDATED
    VALIDATED --> T_PR_STYLED

    ORP_TESTS --> REFUSAL_RATCHET
    OP_TESTS --> REFUSAL_RATCHET
    REFUSAL_RATCHET -->|"no handler found"| T_CI_FAIL
    PALETTE_RATCHET -->|"no palette found"| T_CI_FAIL
    REFUSAL_RATCHET -->|"all callers compliant"| T_CI_PASS
    PALETTE_RATCHET -->|"all generators compliant"| T_CI_PASS

    class ORP,OP handler;
    class SKILL_CALL phase;
    class GATE detector;
    class DISCARD,ALL_REFUSED stateNode;
    class DIAG gap;
    class EMPTY output;
    class EXECUTE,VALIDATE phase;
    class VALIDATED output;
    class REFUSAL_RATCHET,PALETTE_RATCHET newComponent;
    class ORP_TESTS,OP_TESTS handler;
    class T_PR_STYLED,T_PR_CLEAN,T_CI_FAIL,T_CI_PASS terminal;

Loading

Process/Execution Flow Diagram

%%{init: {'flowchart': {'nodeSpacing': 40, 'rankSpacing': 50, 'curve': 'basis'}}}%%
flowchart TB
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef terminal fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;

    START([run_skill open-research-pr / open-pr])

    subgraph LensLoop ["● LENS ITERATION LOOP (Step 4 / Step 5)"]
        direction TB
        ITER["For each lens_slug<br/>━━━━━━━━━━<br/>Iterate exp-lens / arch-lens list"]
        INVOKE["Skill tool call<br/>━━━━━━━━━━<br/>/autoskillit:{lens-slug}"]
        GATED{"Skill tool response<br/>contains 'disable-model-invocation'<br/>or 'cannot be used'?"}
    end

    subgraph RefusalRouting ["● REFUSAL ROUTING (added to both skills)"]
        direction TB
        SKIP["Discard silently<br/>━━━━━━━━━━<br/>do NOT write freehand"]
        MORE{"More lens slugs<br/>remaining?"}
        WRITE_DIAG["Write diagnostic<br/>━━━━━━━━━━<br/>lens_unavailable_{ts}.txt<br/>(open-research-pr only)"]
        SET_EMPTY["validated_diagrams = []<br/>━━━━━━━━━━<br/>Propagate to composition step"]
    end

    subgraph SuccessRouting ["UNMODIFIED SUCCESS ROUTING"]
        direction TB
        EXECUTE["Lens executes<br/>━━━━━━━━━━<br/>Diagram generated<br/>Canonical palette applied"]
        CHECK_MARKERS{"Block contains<br/>★ or ● markers?"}
        APPEND["Append to<br/>validated_diagrams<br/>━━━━━━━━━━<br/>Styled block collected"]
    end

    subgraph Composition ["PR BODY COMPOSITION (Step 6)"]
        direction TB
        DIAGS_EMPTY{"validated_diagrams<br/>empty?"}
        INCLUDE["Include diagram section<br/>━━━━━━━━━━<br/>## Architecture Impact<br/>## Experiment Design"]
        OMIT["Omit section entirely<br/>━━━━━━━━━━<br/>No placeholder in PR body"]
    end

    subgraph CIRatchet ["★ NEW CI CONTRACT RATCHET"]
        direction TB
        DISCOVER["★ Scan skills_extended/<br/>━━━━━━━━━━<br/>Auto-discover sub-skill callers<br/>and diagram generators"]
        CHECK_REFUSAL{"★ All callers have<br/>refusal handler?"}
        CHECK_PALETTE{"★ All generators have<br/>palette or mermaid-load?"}
    end

    T_PR_STYLED([PR with styled diagram section])
    T_PR_CLEAN([PR without diagram section])
    T_CI_PASS([CI PASS])
    T_CI_FAIL([CI FAIL])

    START --> ITER
    ITER --> INVOKE
    INVOKE --> GATED
    GATED -->|"yes — refused"| SKIP
    GATED -->|"no — executes"| EXECUTE
    SKIP --> MORE
    MORE -->|"yes"| ITER
    MORE -->|"no — all refused"| WRITE_DIAG
    WRITE_DIAG --> SET_EMPTY
    EXECUTE --> CHECK_MARKERS
    CHECK_MARKERS -->|"yes — valid"| APPEND
    CHECK_MARKERS -->|"no — invalid"| ITER
    APPEND --> ITER

    SET_EMPTY --> DIAGS_EMPTY
    ITER -->|"loop exhausted"| DIAGS_EMPTY
    DIAGS_EMPTY -->|"yes — empty"| OMIT
    DIAGS_EMPTY -->|"no — has diagrams"| INCLUDE
    INCLUDE --> T_PR_STYLED
    OMIT --> T_PR_CLEAN

    DISCOVER --> CHECK_REFUSAL
    DISCOVER --> CHECK_PALETTE
    CHECK_REFUSAL -->|"all compliant"| T_CI_PASS
    CHECK_REFUSAL -->|"gap found"| T_CI_FAIL
    CHECK_PALETTE -->|"all compliant"| T_CI_PASS
    CHECK_PALETTE -->|"gap found"| T_CI_FAIL

    class START terminal;
    class ITER,INVOKE phase;
    class GATED,MORE,CHECK_MARKERS,DIAGS_EMPTY detector;
    class SKIP,WRITE_DIAG gap;
    class SET_EMPTY,APPEND stateNode;
    class EXECUTE handler;
    class INCLUDE,OMIT output;
    class DISCOVER,CHECK_REFUSAL,CHECK_PALETTE newComponent;
    class T_PR_STYLED,T_PR_CLEAN,T_CI_PASS,T_CI_FAIL terminal;

Loading

Closes #637

Implementation Plan

Plan file: /home/talon/projects/autoskillit-runs/remediation-20260405-212016-569394/.autoskillit/temp/rectify/rectify_sub-skill-refusal-and-palette-contracts_2026-04-05_175800_part_a.md

🤖 Generated with Claude Code via AutoSkillit

Token Usage Summary

Step	uncached	output	cache_read	cache_write	count	time
investigate	21	10.0k	369.4k	48.0k	1	5m 17s
rectify	24	36.2k	646.6k	119.1k	1	13m 56s
dry_walkthrough	41	56.1k	1.7M	134.0k	2	16m 39s
implement	85	36.7k	4.4M	127.9k	2	15m 37s
assess	32	10.9k	1.1M	46.8k	1	5m 48s
open_pr	37	26.4k	1.5M	71.7k	1	8m 29s
Total	240	176.2k	9.7M	547.6k		1h 5m

[Trecek]

AutoSkillit PR Review — Verdict: changes_requested

12 blocking issues found (see inline comments). Running on own PR — using COMMENT instead of REQUEST_CHANGES.

[Trecek]

AutoSkillit review found 12 blocking issues. See inline comments. Note: REQUEST_CHANGES was downgraded to COMMENT due to own-PR restriction — verdict is changes_requested.