CLI Spec Access & Injection — embedded spec tiers, `ail spec` command, native pipeline injection

[AlexChesser]

Plan: CLI Spec Access & Injection

Context

AIL is a YAML-orchestrated pipeline runtime for LLM agents. The spec (~47K words, 37 files, ~330KB, ~82K tokens) is the primary published artifact. A key use case is pipelines that design their own pipelines — an LLM generating AIL YAML. Since no LLM has been trained on AIL, the spec must be injectable at runtime so the LLM can learn the format in-context.

The core problem: How to get AIL's specification into an LLM's context window efficiently, with zero accuracy loss, at minimum token cost?

---

Part 1: Research Findings — Compression Approaches

1.1 Approaches Evaluated and Ruled Out

Approach	Finding	Verdict
AAAK dialect (MemPalace)	Pipe-delimited 11-field schema for entity-relationship data. No slots for rules/syntax/code. 12.4-point accuracy regression vs raw text.	Wrong data model. Not applicable.
TokenCrush	Commercial SaaS API (requires TOKENCRUSH_API_KEY). ML-based, server-side, not deterministic, not self-hostable. Medium article is promotional.	Vendor lock-in. Not usable.
LLMLingua / LLMLingua-2 (Microsoft)	Requires running a small LM (GPT-2/BERT) for token importance scoring. 50-80% compression.	Not deterministic. Requires ML inference.
Binary formats (gzip, protobuf, CBOR)	LLMs process text tokens. Base64 encoding increases token count ~33%.	Physically impossible for LLM consumption.
Marqant (Rust crate)	Custom .mq format. README mentions "quantum compression" and "DNS integration" — red flags.	Experimental/unreliable.

1.2 Viable Deterministic Approaches

Approach	Deterministic?	Compression	Accuracy	Rust-native?	Notes
pulldown-cmark strip + abbreviation dict	Yes	~17% (measured on actual AIL spec)	100% lossless	Yes	Markdown syntax overhead is only 1.7%; biggest win is removing YAML examples (13.8%)
compression-prompt (Rust crate)	Yes	~50%	~91%	Yes	IDF-based statistical filtering. Removes common words. 10.58 MB/s. 350+ validated test pairs. Lossy — risky for normative spec text ("MUST", "SHALL", "isolated").
claw-compactor (Python)	Yes	15-31%	ROUGE-L 0.72 at 0.5	No (Python subprocess)	14-stage pipeline. Lossless stages give ~15-20%. Full pipeline ~31% but lossy.
MetaGlyph (symbol substitution)	Partially	62-81% on rules	Model-dependent (75-98%)	Portable	Replace prose patterns with math symbols (∈, ⇒, ¬). Works for conditional rules, not for prose.
Hand-authored compression	N/A	85-97% (T1: 2-3K, T2: 10-15K vs 82K)	100% (human judgment)	N/A	A human writing compressed tiers preserves accuracy far better than any algorithm.

1.3 Critical Measured Insight

A research agent actually measured compression of the AIL spec files:

• Markdown formatting overhead: only 1.7% — stripping ##, **, |, blank lines etc. barely helps
• Table-to-KV conversion is counterproductive — markdown tables are MORE compact than KEY: VALUE because headers aren't repeated per row
• Abbreviation dictionary (25 words): 2.7% — modest because technical terms are already short
• YAML example removal: 13.8% — the single biggest mechanical win
• Realistic ceiling for all mechanical transforms combined: ~17% (saving ~14K tokens)
• At $3/M input tokens (Claude Sonnet), that's ~$0.04 per context load — the engineering cost of a build-time compressor likely exceeds the savings

1.4 Conclusion: Compression Strategy

The most impactful token savings come from two things that algorithms can't do:

1. Selective section loading — inject only the 2-3 sections relevant to the task (~2-5K tokens each) instead of the full 82K. This is a 90%+ reduction with zero accuracy loss.
2. Hand-authored T1/T2 — a human can compress 82K tokens of spec into a 2-3K schema (T1) or 10-15K compact reference (T2) with far better accuracy than any algorithm.

Therefore:

• T1 (schema): Hand-authored annotated YAML schema. Checked in at spec/compressed/schema.yaml.
• T2 (compact): Hand-authored compressed NL reference. Checked in at spec/compressed/compact.md.
• T3 (prose): Raw concatenation of spec files. Build-time only.
• Per-section access: The most valuable feature — enables surgical injection of only what's needed.
• compression-prompt crate: Expose as an optional --format compressed flag for users who want algorithmic compression and accept ~9% quality risk. Do NOT use for T2.

---

Part 2: Architecture — Standalone ail-spec Crate

2.1 Why a Separate Crate

A new ail-spec/ workspace member isolates:

• Build-time spec processing and embedding
• All static string data (~330KB)
• Optional compression-prompt dependency
• Complexity away from core execution logic

Dependency graph:

ail-spec  →  (no runtime deps; pulldown-cmark + compression-prompt as build-deps)
ail-core  →  ail-spec  (for native context: spec: and append_system_prompt: - spec:)
ail       →  ail-core  (transitively gets ail-spec)

Spec files remain at workspace root (spec/). The ail-spec/build.rs reads ../spec/.

2.2 Build-Time Embedding via build.rs

The build.rs script:

1. Scans ../spec/core/s*.md and ../spec/runner/r*.md
2. For each file: extracts section ID (stem before first -), title (first ## heading), word count
3. Generates embedded_specs.rs with:
   • Each section as a &'static str via include_str!
   • Concatenated T3 constants: FULL_PROSE, CORE_PROSE, RUNNER_PROSE
   • Metadata array: SECTIONS: &[SpecSection]
   • Lookup function: section_content(id) -> Option<&'static str>
4. Embeds T1 from ../spec/compressed/schema.yaml
5. Embeds T2 from ../spec/compressed/compact.md
6. cargo:rerun-if-changed on ../spec/core, ../spec/runner, ../spec/compressed

2.3 Public API

// ail-spec/src/lib.rs

pub struct SpecSection {
    pub id: &'static str,       // "s05", "r02"
    pub title: &'static str,    // "5. Step Specification"
    pub word_count: usize,
}

/// Single section by ID (e.g., "s05", "r02")
pub fn section(id: &str) -> Option<&'static str>;

/// All section metadata
pub fn list_sections() -> &'static [SpecSection];

/// Full concatenated spec (T3, ~82K tokens)
pub fn full_prose() -> &'static str;

/// Core sections only (T3)
pub fn core_prose() -> &'static str;

/// Runner sections only (T3)
pub fn runner_prose() -> &'static str;

/// Compact reference (T2, ~10-15K tokens, hand-authored)
pub fn compact() -> &'static str;

/// Annotated YAML schema (T1, ~2-3K tokens, hand-authored)
pub fn schema() -> &'static str;

---

Part 3: CLI Command — ail spec

ail spec                          # full spec (T3 prose, default)
ail spec --format schema          # T1 annotated YAML schema
ail spec --format compact         # T2 hand-authored compressed reference
ail spec --format prose           # T3 full prose (explicit)
ail spec --section s05            # one section (prose)
ail spec --section s05,s11,r02    # multiple sections
ail spec --list                   # section IDs + titles + word counts
ail spec --core                   # core sections only
ail spec --runner                 # runner sections only

Implementation:

• ail/src/command/spec.rs — SpecCommand struct
• Added to Commands enum in ail/src/cli.rs
• Output to stdout (pipeable, works with context: shell: steps)

---

Part 4: Pipeline Spec Injection — Both Mechanisms

4.1 Native context: spec: Source

pipeline:
  - id: learn_ail
    context:
      spec: compact              # injects T2 as step result

Accepted values: compact | schema | prose | section IDs (s05, r02)

Changes required:

• config/dto.rs: Add spec: Option<String> to ContextDto
• config/domain.rs: Add Spec(SpecQuery) to ContextSource; add SpecQuery enum
• config/validation/step_body.rs: Parse context: spec:, validate value
• executor/dispatch/context.rs: Handle ContextSource::Spec — call ail_spec functions, return as TurnEntry

4.2 Native append_system_prompt: - spec: Source

pipeline:
  - id: design_pipeline
    prompt: "Design an AIL pipeline for: {{ step.invocation.prompt }}"
    append_system_prompt:
      - spec: compact             # inject T2 into system prompt
      - spec: s05                 # inject specific section

Changes required:

• config/dto.rs: Add spec: Option<String> to AppendSystemPromptStructuredDto
• config/domain.rs: Add Spec(SpecQuery) to SystemPromptEntry
• config/validation/system_prompt.rs: Parse and validate spec: entry
• executor/helpers/system_prompt.rs: Resolve SystemPromptEntry::Spec

4.3 Zero-Code Fallback (works with just the CLI)

pipeline:
  - id: learn_ail
    context:
      shell: "ail spec --format compact"

---

Part 5: End-to-End Example — Self-Designing Pipeline

version: "1"
pipeline:
  - id: plan
    prompt: |
      A user wants: {{ step.invocation.prompt }}
      Design an AIL pipeline (.ail.yaml) that accomplishes this.
      Output ONLY the YAML, no explanation.
    append_system_prompt:
      - spec: compact             # T2: enough to write correct YAML
      - spec: s05                 # step specification details
      - spec: s11                 # template variable reference

  - id: validate_output
    context:
      shell: |
        echo '{{ step.plan.response }}' > /tmp/candidate.ail.yaml
        ail validate --pipeline /tmp/candidate.ail.yaml 2>&1

  - id: fix_if_needed
    prompt: |
      The pipeline you generated had validation errors:
      {{ step.validate_output.result }}
      
      Fix the YAML. Output ONLY the corrected YAML.
    condition: "{{ step.validate_output.exit_code }} != 0"
    resume: true
    append_system_prompt:
      - spec: compact

---

Part 6: Files to Create/Modify

New files

File	Purpose
ail-spec/Cargo.toml	Crate manifest (build-dep: pulldown-cmark; no runtime deps)
ail-spec/build.rs	Scan spec files, extract metadata, generate embedding code
ail-spec/src/lib.rs	Public API + include! generated code
ail/src/command/spec.rs	ail spec CLI command
spec/compressed/schema.yaml	T1 hand-authored annotated YAML schema
spec/compressed/compact.md	T2 hand-authored compressed NL reference
spec/core/s31-spec-access.md	New spec section documenting this feature
ail-spec/tests/embedding.rs	Tests for section lookup, tier content, completeness

Modified files

File	Change
Cargo.toml (workspace)	Add ail-spec to members
ail-core/Cargo.toml	Add ail-spec = { path = "../ail-spec" }
ail-core/src/config/dto.rs	Add spec: to ContextDto and AppendSystemPromptStructuredDto
ail-core/src/config/domain.rs	Add Spec(SpecQuery) to ContextSource and SystemPromptEntry; add SpecQuery enum
ail-core/src/config/validation/step_body.rs	Parse context: spec:
ail-core/src/config/validation/system_prompt.rs	Parse append_system_prompt: - spec:
ail-core/src/executor/dispatch/context.rs	Handle ContextSource::Spec
ail-core/src/executor/helpers/system_prompt.rs	Handle SystemPromptEntry::Spec
ail/src/cli.rs	Add Spec variant to Commands enum
ail/src/main.rs	Dispatch Spec command
spec/core/s05-step-specification.md	Document context: spec: and append_system_prompt: - spec:
spec/README.md	Add s31 entry, document tiers and ail spec command

---

Part 7: Verification

1. cargo build succeeds with ail-spec crate
2. ail spec --list prints all 37 section IDs with titles and word counts
3. ail spec --section s05 prints step specification prose verbatim (byte-identical to file)
4. ail spec --format compact prints T2 hand-authored compressed reference
5. ail spec --format schema prints T1 annotated YAML schema
6. ail spec --core prints only s-prefixed sections
7. ail spec --runner prints only r-prefixed sections
8. Pipeline with context: spec: compact produces TurnEntry with T2 content
9. Pipeline with append_system_prompt: - spec: s05 injects section into runner's system prompt
10. Adding a new spec/core/s32-foo.md file automatically appears in ail spec --list on next build (no Rust code changes)
11. cargo nextest run passes all existing + new tests
12. cargo clippy -- -D warnings && cargo fmt --check clean

---

Part 8: Implementation Order

1. Create ail-spec crate scaffold — Cargo.toml, lib.rs, build.rs skeleton
2. Implement build.rs — spec file discovery, metadata extraction, T3 concatenation, include_str! generation
3. Wire public API — lib.rs with section(), list_sections(), full_prose(), etc.
4. Hand-author spec/compressed/schema.yaml (T1 annotated YAML schema, ~2-3K tokens)
5. Hand-author spec/compressed/compact.md (T2 compressed reference, ~10-15K tokens)
6. Add ail-spec tests — section lookup, tier completeness, no empty sections, round-trip
7. Create CLI command — ail/src/command/spec.rs, wire into cli.rs + main.rs
8. Add native context: spec: source — dto, domain, validation, executor changes
9. Add native append_system_prompt: - spec: source — dto, domain, validation, executor
10. Write spec section s31 — document spec access feature
11. Update existing spec sections — s05 (new context/system_prompt source), README