Self-created skills lack mechanism-level guarantees for correctness and execution consistency

[fancpp]

Problem: Self-created skills lack mechanism-level guarantees for correctness and execution consistency

Background

Hermes Agent has a powerful skill auto-creation loop — after complex or error-prone tasks, the agent can persist its approach as a reusable skill. This is one of the system's strongest differentiators.

However, the current mechanism has a structural defect: the agent is simultaneously the author, executor, and quality inspector of its own skills. There is no external validation point or consistency check, which creates several failure classes that prompt-level mitigations cannot fully address.

Related Prior Issues

Issue	Focus	Status
#6051	Auto-creation learns transient failures → persistent tool avoidance ("learned helplessness")	✅ Fixed (PR #23004) — prompt-level fix telling reviewer not to persist env-dependent failures
#416	No automated quality checks on create/edit (YAML linting, file existence)	❌ Open
#7816	Skill lifecycle: usage metadata, staleness, archival	⚡ Partially landed (Curator)
#16077	RFC: Curator — background skill maintenance	✅ Implemented

What Remains Unaddressed

The existing fixes share a common pattern: they are prompt-level or post-hoc mitigations, not mechanism-level guarantees.

1. Skill auto-creation learns from transient failures, causing persistent tool avoidance (learned helplessness) #6051's fix is a prompt adjustment — it tells the background reviewer "don't capture transient failures." But it doesn't prevent the agent from encoding suboptimal approaches, incorrect workflows, or session-specific lucky paths as durable skills.

2. There is no reproducibility check — When a skill is created, there is no automated step like "run this skill's instructions in an isolated context once to verify they produce the claimed result."

3. There is no cross-session consistency guarantee — Running the same skill twice on the same input can produce different results (model temperature, context variation, tool availability), but there is no mechanism to detect, quantify, or flag this drift.

4. Self-validation problem — The same model that writes a skill is also the sole judge of its correctness. Without an external validation step (e.g., secondary model review, isolated dry-run), blind spots are structurally invisible.

Proposed Design Directions

We should consider adding mechanism-level guardrails — not just better prompts:

1. Reproducibility Verification on Create

When a skill is created, if its instructions include a runnable workflow (commands, API calls, code), the system should:

• Automatically execute it in an isolated context (or simulate a fresh session)
• Compare actual output against the claimed behavior
• Tag the skill as verified: true/false based on the outcome
• Block or warn when verification fails

Scope: Only for skills with executable content. Pure knowledge/description skills would be exempt.

2. Consistency Metadata

Each skill should carry runtime metadata that anchors it to the execution context in which it was created:

# Proposed sidecar fields (in .usage.json or frontmatter, not just prompt text)
runtime:
  model_created: "claude-sonnet-4-20250514"
  model_version: "..."          # model hash / version if available
  execution_count: 0
  success_rate: 0.0             # % of recent runs that completed without error
  last_verified_with_model: "claude-sonnet-4-20250514"
  last_verified_at: null        # ISO timestamp of last successful re-verification
  consistency_score: null       # similarity score between 2+ runs of same task

This allows the system (and Curator) to make informed decisions:

• A skill with success_rate: 0.3 gets lower priority in prompt injection
• A skill from a different model version can be flagged for re-verification when the model changes
• A skill never re-verified after creation can be tagged as experimental

3. Secondary-Review Gate

Before a newly created skill is persisted to ~/.hermes/skills/ and becomes discoverable, run a lightweight secondary review:

• Parse any tool references and validate against the live tool registry (partial: #416)
• Detect TODO, FIXME, placeholder values in the skill body
• Flag skills whose instructions reference tools/endpoints that don't exist
• Reject skills that encode negative claims about tool availability

This is a write-time gate, not a prompt tweak — a hard check before the skill enters the system.

4. Cross-Run Consistency Scoring (Optional / Advanced)

For skills that produce deterministic outputs (code generation, data transformation), periodically execute the same task twice and compare:

• Output structure / schema
• Error rate
• Behavior differences

If divergence exceeds a threshold, downgrade the skill's confidence level or notify the user.

Why Prompt-Level Fixes Aren't Enough

The #6051 fix works for the specific case of "transient env failure → negative constraint." But it's a software patch on top of a structural gap:

• Prompt instructions can drift, be overridden, or be ignored by future prompt changes
• The reviewer model itself has no way to know whether its instructions are correct — it only knows what the prompt told it
• The underlying mechanism (write skill → use skill → no re-validation) is unchanged

Mechanism-level fixes, by contrast, are:

• Model-independent — they work regardless of which model writes or reviews the skill
• Self-correcting — a verification failure is visible and actionable, not silently absorbed
• Auditable — the system can report "skill X has never been verified" or "skill Y has inconsistent outputs"

Concrete Failure Scenario (Not Covered by Existing Fixes)

1. User asks for "generate a data processing pipeline"
2. Agent writes a pipeline that works in the current session but uses a deprecated API endpoint
3. Agent creates skill data-pipeline-generator encoding the deprecated approach
4. Two weeks later, user asks the same question
5. Agent loads the skill, generates code with the now-broken API endpoint
6. Failure — but no mechanism detects that the skill itself is the root cause

Without mechanism-level verification, this scenario is invisible to both the agent and the user.

Related

• #10666 — Skill quality tiers (core / recommended / experimental). A verification_status field could feed directly into tier assignment.
• #7816 — Curator lifecycle. Verification metadata would give the Curator concrete signals for consolidation and archival decisions.

[alt-glitch]

This is closely related to #416 (Skill Validation & Linting — Automated Quality Checks on Skill Create/Edit), which remains open and requests similar mechanism-level validation at create/edit time. The scope here is broader (reproducibility checks, isolation testing, not just YAML linting), so keeping this open as a superset. See also #6051 (fixed via #23004) and #7816 (partially landed via Curator).

[fancpp]

Problem

Hermes Agent's skill auto-creation system has a structural defect: the agent serves simultaneously as the author, executor, and quality inspector of its own skills. There is no mechanism-level validation that a skill's instructions are correct, reproducible, or consistent across executions.

What Prior Work Has Covered

Issue	Focus	What Changed
#6051	Auto-creation learns transient failures → "learned helplessness"	Prompt fix: reviewer told not to persist env-dependent failures (PR #23004)
#416	No format-level validation on create/edit	Not yet implemented — would add YAML linting, file existence checks
#7816	Skill lifecycle: usage metadata, staleness, archival	Landed in #17213/#17277 — sidecar metadata, Curator state machine
#16077	RFC: Curator — background skill maintenance	Implemented

All prior fixes operate at the prompt or post-hoc level — they don't change the fundamental gap: skills enter the system with no correctness guarantee, and there is no mechanism to detect when they drift or become wrong.

---

Proposed Solution

Overview

Add a verification layer to the skill lifecycle with three mechanisms:

Creation ──→ Verification Gate ──→ Active ──→ Periodic Re-verification ──→ Stale/Archive
                 │                                    │
                 └── Block on failure                 └── Downgrade on persistent failure

The key invariant: every agent-created skill must pass a structured verification step before it becomes visible and usable.

---

Phase 1: On-Create Verification Gate (Write-Time)

Location: tools/skill_manager_tool.py — in _create_skill() and _edit_skill(), after the security scan but before returning success.

Goal: Catch obviously wrong skills before they enter the system.

1a. Tool/Command Reference Validation

Parse the skill body for references to tools and external commands, then validate them against the live registry:

def validate_skill_references(content: str) -> List[str]:
    """Check tool/command references in skill content. Returns list of warnings."""
    from tools.registry import registry
    
    warnings = []
    
    # Find tool references: backticked names, "use X tool" patterns
    tool_refs = extract_tool_references(content)
    for ref in tool_refs:
        if not registry.has(ref):
            warnings.append(f"References tool '{ref}' which is not registered")
    
    # Find command references: `command --flag` patterns
    cmd_refs = extract_command_references(content)
    for cmd in cmd_refs:
        if not shutil.which(cmd):
            warnings.append(f"References command '{cmd}' which is not available on PATH")
    
    return warnings

Behavior:

• Warnings are returned in the skill_manage response
• Non-blocking — the skill is still created, but the agent is told about the issues
• Future enhancement: make blocking configurable via skills.strict_reference_validation

1b. Negative Claims Detection

Scan for patterns that encode environment-specific or permanent-sounding negative claims:

_NEGATIVE_PATTERNS = [
    r"(?i)\b(does not|don't|won't|cannot|can't)\b.*\b(work|function|run|install)\b",
    r"(?i)\btool\b.*\b(unavailable|broken|fails)\b",
    r"(?i)\b(not possible|cannot be done)\b",
]

Behavior:

• Match → skill is tagged as verification_warnings: ["contains negative tool claims"] in .usage.json
• The agent's prompt injection layer deprioritizes skills with active warnings
• This is the mechanism-level fix for Skill auto-creation learns from transient failures, causing persistent tool avoidance (learned helplessness) #6051 — not telling the reviewer "don't write negatives," but detecting them at write time and acting on them structurally

1c. Template/Placeholder Detection

Detect obviously incomplete content:

_TEMPLATE_PATTERNS = [
    r"(?i)\bTODO\b",
    r"(?i)\bFIXME\b",
    r"(?i)\bplaceholder\b",
    r"\{\{[^}]+\}\}",        # template variables like {{api_key}}
    r"<[A-Z_]+>",            # placeholder like <YOUR_API_KEY>
]

Behavior:

• Warnings only — the skill is created but the agent is nudged to fill in placeholders
• Similar to how linters flag TODOs in code

---

Phase 2: Sidecar Verification Metadata (Provenance Extension)

Location: tools/skill_usage.py — extend the _empty_record() dict.

Current record:

{
    "created_by": None,
    "use_count": 0,
    "view_count": 0,
    "last_used_at": None,
    "last_viewed_at": None,
    "patch_count": 0,
    "last_patched_at": None,
    "created_at": _now_iso(),
    "state": STATE_ACTIVE,
    "pinned": False,
    "archived_at": None,
}

Proposed additions:

{
    # ... existing fields ...
    
    # === Verification metadata ===
    "verification_status": "unverified",   # "unverified" | "verified" | "failed" | "degraded"
    "verifications_passed": 0,             # count of successful verifications
    "verifications_failed": 0,             # count of failed verifications
    "last_verified_at": None,              # ISO timestamp
    "last_verification_result": None,      # summary of last check
    
    # === Model/context provenance ===
    "model_created_by": None,              # "claude-sonnet-4-20250514"
    "model_version": None,                 # model hash if available
    
    # === Warning flags (from Phase 1) ===
    "verification_warnings": [],           # list of active warnings (unmet prereqs, negative claims, etc.)
    
    # === Cross-run consistency ===
    "consistency_score": None,             # float 0.0–1.0, null = never measured
    "consistency_last_checked": None,      # ISO timestamp
}

Key design decisions:

• Sidecar (not frontmatter) — keeps operational data out of user-authored content, consistent with existing pattern
• All fields are best-effort (counters log at DEBUG, never block tool execution)
• created_by already exists and is set by mark_agent_created() — model provenance is a natural extension

---

Phase 3: Verification Tool (Agent-Facing verify_skill)

Location: New file tools/skill_verifier.py, registered via registry.register().

Purpose: A tool the agent can call to re-verify a skill after patching it, or that the Curator can invoke during maintenance.

Schema:

registry.register(
    name="verify_skill",
    toolset="skills",
    schema={
        "name": "verify_skill",
        "description": "Verify that a skill's instructions are self-consistent, its referenced tools exist, and it does not contain stale claims. Updates .usage.json verification metadata.",
        "parameters": {
            "type": "object",
            "properties": {
                "skill_name": {
                    "type": "string",
                    "description": "Name of the skill to verify"
                }
            },
            "required": ["skill_name"]
        }
    },
    handler=lambda args, **kw: verify_skill(args.get("skill_name", ""), task_id=kw.get("task_id"))
)

What it checks:

1. Frontmatter structural validity — delegates to existing _validate_frontmatter()
2. Tool reference resolution — same as Phase 1a
3. Command availability — same as Phase 1a
4. File integrity — all referenced files in references/, templates/, scripts/ exist
5. No stale/contradictory claims — Phase 1b negative pattern scan
6. No placeholder content — Phase 1c template pattern scan

Output:

{
    "success": True,
    "skill": "my-skill",
    "status": "verified",         # "verified" | "failed" | "degraded"
    "issues": [],                 # list of findings
    "warnings": [],               # non-blocking concerns
    "passed": 5,                  # count of checks passed
    "failed": 0,                  # count of checks failed
}

How .usage.json gets updated:

• All checks pass → verification_status: "verified", verifications_passed++, last_verified_at = now
• Any check fails → verification_status: "failed", verifications_failed++, last_verified_at = now
• Mixed (warnings only) → verification_status: "degraded", last_verified_at = now

---

Phase 4: Curator Integration (Periodic Re-Verification)

Location: agent/curator.py — extend the review pass.

Current curator flow:

1. Load usage data → identify stale/archivable skills → auto-transition
2. Spawn review agent for consolidation/archival decisions

Proposed additions:

1. Pre-review scan: Before spawning the review agent, run:

   • For skills with verification_status = "unverified" and age > 7 days → tag as stale
   • For skills with verifications_failed >= 3 since last successful → auto-archive
   • For skills with consistency_score < 0.5 (if measured) → tag as stale

2. Review agent context: Include verification history in the review prompt so the agent knows which skills have known issues.

3. Configurable thresholds:

curator:
  enabled: true
  interval_hours: 168           # 7 days (unchanged)
  stale_after_days: 30          # (unchanged)
  archive_after_days: 90        # (unchanged)
  verification:
    max_failures_before_archive: 3
    max_age_unverified_days: 7
    enable_reverify: true       # Phase 4 gate
    enable_consistency_check: false  # Phase 5 gate

---

Phase 5: Cross-Run Consistency Checking (Optional / Future)

Goal: Detect when the same skill produces different results across runs.

Approach: For skills with deterministic output (code generation templates, data processing pipelines), define an optional test harness:

# In SKILL.md frontmatter (optional)
verification:
  test_command: "python -c \"from my_skill import process; print(process('test_input'))\""
  expected_output_contains: "expected string"
  timeout_seconds: 30

When test_command is present and verification.enable_consistency_check is on:

• Execute the test command in a subprocess
• Compare output against expected_output_contains
• Record pass/fail in .usage.json consistency_score
• If 3 consecutive runs fail → verification_status: "degraded"

Why optional: Most skills are procedural instructions, not runnable code. The test harness should be authored intentionally, not auto-generated.

---

Migration / Backward Compatibility

All existing skills start with:

{
    "verification_status": "unverified",  # Phase 2 default
    "model_created_by": None,
    "verifications_passed": 0,
    "verifications_failed": 0,
}

On first verify_skill call, they get a real status. Existing behavior is fully preserved — skills with no verification metadata are treated as before (no downgrading).

---

Summary of Changes by File

File	Change
tools/skill_usage.py	Extend _empty_record() with verification fields; add mark_verified() / mark_failed() functions
tools/skill_manager_tool.py	Add tool/command reference validation in _create_skill(); add negative-claim and placeholder detection
tools/skill_verifier.py	New file — verify_skill tool for agent-facing on-demand verification
tools/skills_tool.py	Surface verification_status in skills_list output; deprioritize skills with active warnings
agent/curator.py	Pre-review scan for verification failures; auto-archive on threshold; include verification history in review prompt
toolsets.py	Add skill_verifier to _HERMES_CORE_TOOLS
hermes_cli/config.py	Add curator.verification.* config defaults

Why This Closes the Gap

1. Model-independent — Works regardless of which model writes or reviews the skill
2. Self-correcting — A verification failure is visible and actionable, not silently absorbed
3. Auditable — The system can report "skill X has never been verified" or "skill Y has inconsistent outputs"
4. Backward compatible — All existing skills and behavior preserved; no breaking changes
5. Incremental — Each phase can be shipped independently; Phase 1+2 are sufficient to address the core problem
6. Reuses existing infrastructure — Curator state machine, .usage.json sidecar, security scan pattern all exist

[fancpp]

Follow-up: Related Issues & Design Alternatives After Community Feedback

Thanks @alt-glitch for pointing out the connections.

After deeper investigation of the existing issue landscape, here's what I found and how the proposal should be refined.

---

Closely Related Issues Discovered

Issue	Relevance	Key Difference from This Proposal
#416 (teknium1)	Structural validation at create/edit time — YAML linting, file existence checks	Narrower scope (format-level only, no provenance, no re-verification lifecycle). Clean superset relationship
#429 (teknium1)	Skill quality — better descriptions, proactive post-use improvement loop, writing principles	Complements this issue. #429 says "write better skills." This issue says "prove skills are correct." Independent implementation
#13534 (fengcwf)	Pre-creation validation, conflict/dedup detection, usage tracking after 146+ skills in production	Strong overlap on pre-creation validation and usage tracking. Proposes TF-IDF/embedding-based dedup and hermes skills audit — not covered in original proposal
#488 (teknium1)	Zeroshot — blind validation via independent sub-agents	Introduces blind validation concept: validator doesn't see worker's reasoning, only ground truth. Stronger pattern than Phase 1-3 of the current proposal
#406 (teknium1)	Independent code verification & quality gates (inspired by Nightwire)	Baseline-relative quality gates, fail-closed verification, auto-fix loop. Relevant for consistency checking (Phase 5)

---

Two Critical Gaps in the Original Proposal

1. Description overlap / dedup (from #13534)

When skill_manage(action='create') is called, there's no check for whether the description substantially overlaps with an existing skill. In a production environment with 146+ skills (#13534's setup), this creates duplicates the agent can't detect on its own — and each duplicate linearly increases token cost (~4,400 tokens/turn at that scale).

Fix: Add a pre-creation similarity gate to the verification gate. Compare new skill description against all existing skill descriptions using TF-IDF cosine similarity (cheap, no ML dependency). If similarity > threshold (configurable, default ~0.75), warn the agent and suggest a patch instead of a new skill.

2. Self-citation / echo-chamber validation (from #488)

Phase 1-3 of the original proposal still lets the same model validate its own work. The deterministic checks (tool references, file integrity, pattern matching) are objective and fine. But the Curator's consolidation/archival review (Phase 4) uses the same agent model to judge skill quality — which is the self-validation problem restated at a higher level.

Fix: The curator's review pass should use the secondary model (or a configurable alternative provider) when available, not the main session model. The Curator already uses AIAgent(max_iterations=8); adding a model parameter to that fork is minimal plumbing.

---

Better Design Alternatives

Alternative A: "Hard Gate + Soft Signal" (Recommended over original Phase 1 gates)

Instead of making the verification gate behavior configurable, separate the concerns permanently:

• Hard gate (always on): Structural checks only — invalid YAML, missing required fields, broken symlinks, non-existent file references. Block creation if they fail. This covers Feature: Skill Validation & Linting — Automated Quality Checks on Skill Create/Edit #416 and [Feature]: Skill Management — Usage Tracking, Conflict Detection, and Pre-Creation Validation #13534's pre-creation validation in one pass.

• Soft signal (always on): Subjective findings — reference-to-nonexistent-tool (if non-blocking), negative claims, placeholders, description overlap. Written to .usage.json as verification_warnings. The prompt injection layer deprioritizes skills with active warnings. Self-correcting: if the agent fixes warnings and calls verify_skill, the warnings clear.

No skills.strict_reference_validation config flag needed. The distinction is hardcoded: objective failures block, subjective findings signal.

Alternative B: Verification as a Lifecycle State (Simpler than Phase 2+4)

Instead of 7 new .usage.json fields and a separate Curator review pass, treat verification_status as a first-class lifecycle state alongside the existing active/stale/archived:

active(verified) → active(degraded, on warning) → stale → archived

The Curator already has apply_automatic_transitions(). Extend it to read verification_status:

def _apply_verification_transitions():
    for skill, rec in agent_created():
        if rec["verification_status"] == "failed" and rec["verifications_failed"] >= 3:
            _stagger_archive(skill)
        elif rec["verification_status"] == "unverified" and age > 7:
            # Don't archive — just lower confidence weight for prompt injection
            rec["confidence_weight"] = 0.5

Minimal new .usage.json fields:

{
    "verification_status": "unverified",  # unverified | verified | degraded | failed
    "verifications_passed": 0,
    "verifications_failed": 0,
    "last_verified_at": None,
    "model_created_by": None,
}

verification_warnings, consistency_score, model_version deferred until concrete use case.

Alternative C: Remote Verification (Future / Note)

For production setups (multi-agent, 146+ skills), deterministic checks run locally. Cross-run consistency and negative-claim revalidation could optionally delegate to a remote verifier:

curator:
  verification:
    mode: remote       # "local" (default) or "remote"
    endpoint: ""

Keeps the core codebase simple while allowing advanced users to plug in dedicated judge model / replay sandbox. Out of scope for a first PR but worth noting for long-term design.

---

Revised Recommended Scope for a Single PR

Phase A — Structural gate (tools/skill_manager_tool.py, ~150 lines):

• Tool reference validity (registry lookup)
• Command availability (shutil.which)
• Negative claim detection (pattern match)
• Placeholder content detection
• Description overlap warning (TF-IDF, from [Feature]: Skill Management — Usage Tracking, Conflict Detection, and Pre-Creation Validation #13534)
• Hard block on structural failures, soft warning on subjective findings

Phase B — Provenance + state (tools/skill_usage.py, ~80 lines):

• Extend _empty_record(): verification_status, verifications_passed, verifications_failed, last_verified_at, model_created_by
• New: mark_verified() / mark_failed() functions
• Wire model_info through mark_agent_created()

Phase C — Tool + Curator integration (~200 lines):

• New tools/skill_verifier.py: verify_skill agent-facing tool
• agent/curator.py: extend apply_automatic_transitions() to read verification_status, archive on threshold failures

~430 lines across 4 files, zero new dependencies. If #416 lands first, Phase A shrinks ~150 lines since the reference validation logic is shared.

[jax-0n-git]

Concrete repro of the structural defect (2026-05-15, Hermes v0.13.0 / 2026.5.7)

Hit this twice in one session. Both contaminations came from the _spawn_background_review path — banner: 💾 Self-improvement review: Skill 'X' created.

Round 1 — hermes-agent-automation (994 bytes):
The chat-agent's filesystem tool returned empty results for a real, populated directory (transient resolution issue). The agent interpreted "empty tool output" as "empty filesystem" and proposed to inject template files. The user refused the proposal and provided ground truth contradicting the empty-state claim. The background reviewer then auto-wrote a skill canonicalizing the hallucinated empty-directory finding as a durable lesson, with a "pitfall" like "A job ID may exist and an output directory may be present, but if the workload has not executed with tool-access enabled, the directory will be empty." The actual directories had 19 subdirs / 294 files in one of them. Skill was deleted manually.

Round 2 — hyperframes-wrapper-synthesis (2307 bytes):
A 4-round iterative draft of a no_agent cron wrapper. In rounds 1–3 the chat-agent emitted hallucinated CLI commands (hermes agent execute, hermes deliver). The user corrected each round; the final delivered wrapper did NOT contain those calls. The background reviewer then auto-wrote a skill embedding hermes deliver --target <id> as a "Sovereign Wrapper Standard", plus the assertion "the platform's auto-delivery is bypassed in sovereign mode" — the opposite of true (no_agent stdout-to-Telegram is automatic per astrology-daily.sh line 4). The reviewer canonicalized the corrected-and-discarded hallucinated version, not the final corrected version.

Key match for this issue's framing: the reviewer reads transcript-level context, not final-artifact state. So errors that get corrected mid-session can persist as durable canonicalized "lessons." The self-validation problem (#author == #executor == #inspector) shows up not just in the original write but in the review path's choice of what to encode — neither the chat-agent nor the reviewer can tell which parts of a transcript represent settled truth versus retracted error.

Re #21315: Tried hermes --profile <p> config set skills.evolution_mode confirm to at least gate background-review writes per that issue's confirm mode. The key isn't present in v0.13.0 (2026.5.7) or in a fresh ~/hermes-scratch/hermes-agent/ main checkout (grep returned zero evolution_mode references). The merged PR appears to be in the 203 commits between our snapshot and current main. Both occurrences had to be cleaned up manually with rm -rf + gateway bounce in the interim.

One concrete proposal direction in #25833 that would have caught both cases: secondary-review of any hermes <subcommand> or ollama <subcommand> strings in the proposed skill body against the live --help subcommand list — both contaminated skills referenced subcommands that don't exist. That's a deterministic check, no LLM-secondary-reviewer required, no false-positive risk.

Happy to share full chat-transcript and the two deleted skills' verbatim content if useful for repro fixture work.

[hanzckernel]

Additional failure mode: semantic over-generalization from single-session observations

I want to add a concrete subcase that is not fully covered by structural validation (#416), security scanning (#16461), or approval gating (#26298 / #26547): self-created skills can turn a local observation into a global operational rule.

Recent real-world example:

• During local dev-server troubleshooting, Hermes observed project-specific port bindings that were true for that session, e.g. a Vite-style frontend port around 5173 and a Hermes/WUI-adjacent backend/API port around 8648.
• The skill-writing path treated those observed values as a reusable DevServer convention.
• In later sessions, the resulting skill biased the agent toward assuming all similar DevServer setups use those fixed ports.
• That is wrong: ports are environment/project/runtime facts and should be discovered from the active repo/process state, not learned as global constants.

The correct durable lesson is not “DevServer uses ports 5173 and 8648.” It is closer to:

When diagnosing a dev server, discover the actual bind ports from the current source of truth: package scripts, config/env files, startup logs, lsof/ss/netstat, and health probes. Treat historical ports such as 5173 or 864x only as search hints for specific stacks, never as defaults.

Why this matters for this issue:

• The skill can be syntactically valid and structurally well-formed.
• It can pass a security scanner.
• It can even be useful in the original session.
• The defect is epistemic: the system fails to preserve the boundary between observed fact, local convention, and portable rule.

This suggests a narrower mechanism-level gate worth considering alongside the broader reproducibility proposal:

1. Magic-constant detection on background-review/curator skill_manage(create|edit|patch) output:

   • fixed ports (5173, 8648, etc.)
   • absolute paths / usernames
   • branch names / PR numbers / issue numbers / commit SHAs
   • provider/model IDs
   • service names or local process labels

2. Scope/evidence requirement when such constants appear:

   • Where did this value come from?
   • Is it repo-specific, OS-specific, provider-specific, version-specific, or only a historical example?
   • What is the discovery procedure for the next run?

3. Discovery-first rewrite rule:

   • If a value can be retrieved from the live environment, the skill should preserve the lookup procedure rather than the value as a default.
   • Example: “inspect package.json + logs + lsof” instead of “use port 5173”.

4. Negative applicability / anti-overgeneralization section:

   • Require When NOT to use / do_not_use_when for background-created skills that encode operational heuristics.
   • This overlaps with the direction in Add skill-compounding gate to curator (pre-save + dedup + negative examples) #23104, but the port-binding case shows the gate should catch hardcoded environmental constants, not just skill sprawl or missing negative examples.

5. Configurable enforcement mode:

   • warn: include warnings in skill_manage result / background review summary
   • approval: require user review before persisting flagged skill changes
   • block: refuse persistence until the skill states scope, evidence, and discovery steps

Related existing work as I understand it:

• Feature: Skill Validation & Linting — Automated Quality Checks on Skill Create/Edit #416 / feat(skills): add 'hermes skills validate' static lint command #25534: structural/static validation. Useful, but this is semantic/epistemic quality.
• Feature: Skill Lifecycle Quality — Better Descriptions, Proactive Improvement Loop, and Writing Principles (inspired by Anthropic skill-creator) #429: better skill-writing principles. Complementary, but prompt guidance alone may not catch this reliably.
• Add skill-compounding gate to curator (pre-save + dedup + negative examples) #23104: pre-save gate + negative examples for curator-created skills. Very relevant; this comment argues for an explicit “hardcoded environment constant → require scope/discovery” rule.
• [Feature]: skill creation should ask user for confirmation — especially when a同名 skill already exists #26298 / fix(skills): gate background review writes behind approval #26547: approval gating. Helpful for visibility, but it still relies on a human noticing the bad generalization unless the system flags it.
• Skill auto-creation learns from transient failures, causing persistent tool avoidance (learned helplessness) #6051 / fix(review): tell background reviewer not to capture transient env failures as skills #23004: prevents learned helplessness from transient failures. This is the sibling failure mode: learned overconfidence from transient/local successes.

Acceptance criterion for this subcase could be simple:

A background-created skill that mentions fixed port numbers as defaults for a generic dev-server workflow should be flagged unless it either scopes them to a specific project/source or rewrites them as historical hints plus a live discovery procedure.