pi-inline-format

Strict Pi extension project scaffold with:

• a TypeScript Pi extension entrypoint in extensions/
• a Rust core subproject in rust/

Goals

• Follow Pi package and extension conventions
• Keep the Pi-facing wrapper small and type-safe
• Keep parsing/rendering logic in a stricter Rust core
• Enforce strict checks separately for TypeScript and Rust

Layout

pi-inline-format/
├── extensions/        # Pi extension entrypoint(s)
├── rust/              # Rust core subproject
├── .ralphi/           # Ralph loop config/runtime files
├── eslint.config.mjs
├── package.json
└── tsconfig.json

TypeScript commands

npm run typecheck
npm run lint
npm run format

Rust commands

npm run rust:fmt
npm run rust:check
npm run rust:lint
npm run rust:test

Combined check

npm run check

CI contract

The GitHub Actions check workflow is the repository-level guardrail for the package-backed setup.

It currently runs four layers:

1. npm run verify:host-source-upgrade-path
   • explicitly rehearses the reversible local-path to pinned-git migration flow,
   • proves the pinned host source still loads,
   • proves deterministic compare still works after the source switch,
   • restores .pi/settings.json automatically,
   • in CI, runs after the workflow materializes the legacy ../pi-inline-format-extensions sibling checkout expected by the rehearsal.
2. npm run check:pinned-host-runtime
   • covers the headless deterministic regression path for the pinned host source.
3. npm run check:ansi-capture-proof
   • runs the tmux-based ANSI capture proof harness,
   • consumes the new replay.ansi and observer.capture artifacts,
   • verifies that color-sensitive proof survives the ANSI-preserving tmux capture-pane -e bridge for python, javascript, typescript, and bash.
4. npm run check:core
   • covers linting, formatting, TypeScript checks, and all Rust checks/tests.

Use this split to interpret failures:

• upgrade-path rehearsal failures usually indicate package-source or migration regressions,
• check:pinned-host-runtime failures usually indicate package-backed host/runtime regressions before TUI/tmux proof is involved,
• check:ansi-capture-proof failures usually indicate the ANSI-preserving proof bridge or color-sensitive transcript evidence regressed,
• check:core failures usually indicate repo-local diagnostics or Rust/tooling issues rather than package-source wiring.

Rust transcript analysis contract

The Rust CLI reads raw transcript text from stdin and prints JSON shaped like:

{
  "regions": [
    {
      "id": "outer-0",
      "role": "outer",
      "language": "bash",
      "start_byte": 0,
      "end_byte": 18
    },
    {
      "id": "embedded-0",
      "role": "embedded",
      "language": "python",
      "start_byte": 18,
      "end_byte": 30
    },
    {
      "id": "outer-1",
      "role": "outer",
      "language": "bash",
      "start_byte": 30,
      "end_byte": 44
    }
  ],
  "render_blocks": [
    {
      "id": "outer-0",
      "role": "outer",
      "language": "bash",
      "content": "$ python - <<'PY'\n"
    },
    {
      "id": "embedded-0",
      "role": "embedded",
      "language": "python",
      "content": "print('hi')\n"
    },
    {
      "id": "outer-1",
      "role": "outer",
      "language": "bash",
      "content": "PY\n$ echo done\n"
    }
  ]
}

Contract notes:

• regions is ordered and generic so later iterations can add more nested-language patterns.
• render_blocks is the render-ready companion to regions; each block keeps a renderer's language choice separate from the original byte offsets.
• role distinguishes outer transcript content from embedded code content.
• language identifies the rendering language for each region or block.
• start_byte and end_byte preserve original transcript boundaries for later extraction/rendering work.
• The Rust core now drives detection through a generic NESTED_REGION_PATTERNS table, so future nested-language additions can plug in new pattern descriptors without changing the JSON contract shape.
• For python - <<'PY' ... PY, the embedded Python region excludes the heredoc opener and terminator so wrapper content stays in outer bash regions while render_blocks expose a distinct Python block for display.
• The first embedded-language detection case is python - <<'PY' ... PY inside a bash transcript.
• The same PY heredoc split also supports plain bash file-writing flows such as cat > /tmp/delete.me.py <<'PY' ... PY, which is the canonical normal-flow validation prompt for this repo.

TypeScript wrapper integration

The Pi extension keeps TypeScript Pi-facing and delegates transcript analysis to extensions/rust-cli.ts. That helper:

• prefers a fresh Rust CLI binary from rust/target/... when one is newer than the Rust sources,
• falls back to cargo run --manifest-path rust/Cargo.toml --quiet --bin pi-inline-format-core when no fresh binary is available,
• sends the raw transcript to stdin,
• validates the JSON response shape before returning it to the Pi-facing layer,
• includes the invoked command in thrown errors so integration failures are easier to debug.

Deterministic compare helpers

Deterministic compare for live tmux A/B checks is owned by the package-backed host runtime loaded from .pi/settings.json, not by repo-local provider files.

Current deterministic provider surface:

• Provider: inline-deterministic
• Scenario models:
  • canonical-heredoc-compare — Python
  • javascript-heredoc-compare — JavaScript
  • typescript-heredoc-compare — TypeScript
  • bash-heredoc-compare — shell/bash
• Default prompt/model pair:
  • prompt: Use bash to run python from a heredoc with python3. Use PY as the heredoc delimiter exactly. Keep the transcript inline and normal.
  • model: inline-deterministic/canonical-heredoc-compare
• Commands:
  • /inline-format-use-deterministic-model [scenario] — switches the current session to the package-backed deterministic compare model for python, javascript, typescript, or bash.
  • /inline-format-run-deterministic-compare [scenario] — switches to the requested deterministic scenario and submits its matching prompt with no real LLM call.
  • /inline-format-deterministic-status — shows the provider, available scenarios/models, default prompt, and helper commands.

Verification in this repo:

• npm run check:pinned-host-runtime
  • makes sure the pinned public package still serves the shipped Python, JavaScript, TypeScript, and bash proof paths.
• npm run verify:multilanguage-local-package-proof
  • temporarily switches .pi/settings.json to the local root package source ../../pi-inline-format-extensions,
  • checks the Python/JavaScript/TypeScript/bash scenarios,
  • and switches back to the pinned git source automatically.
• npm run smoke:javascript-highlight-compare
  • opens a tmux session with the same JavaScript heredoc on both sides,
  • left pane = the current pinned package,
  • right pane = your local checkout,
  • keeps the normal bash tool row,
  • keeps the same source text,
  • and shows whether the local version changes highlighting only.
• npm run smoke:typescript-highlight-compare
  • does the same side-by-side check for the TypeScript heredoc scenario.

Extra inspection commands exposed by the pinned host package:

• /inline-format-intel-status — shows which language backends are available.
• /inline-format-inspect-sample <scenario> — inspects a built-in sample heredoc.
• /inline-format-explain-symbol <scenario> <symbol> — explains a symbol in a sample heredoc.
• /inline-format-find-definition <scenario> <symbol> — finds where a symbol comes from in a sample heredoc.
• /inline-format-highlight-symbol <scenario> <symbol> — shows matching symbol ranges in a sample heredoc.
• /inline-format-semantic-tokens <scenario> — shows the raw token data for JS/TS.
• /inline-format-diagnostics-sample <scenario> — shows diagnostics for a sample heredoc.

Pi-facing entrypoints exposed from extensions/index.ts:

• /inline-format-status — reports that the project-local diagnostics wrapper is wired to the Rust CLI.
• /inline-format-analyze — analyzes either the provided transcript argument or a built-in heredoc sample.
• /inline-format-render — renders the transcript as distinct language-aware markdown code fences derived from Rust render_blocks.
• analyze_inline_transcript — Pi tool that returns the full Rust JSON contract for a raw transcript.
• render_inline_transcript — Pi tool that returns markdown-ready output plus structured render_blocks for downstream rendering.
• package-backed host runtime seams — loaded from .pi/settings.json, currently owning the built-in bash override, deterministic compare helpers, and summary suppression behavior for the normal Pi user flow.

Heredoc language support

Bash heredocs can hold almost any text. This table is about what this package knows how to recognize and highlight today.

What kind of highlighting do you get?

• Basic highlighting = the package detects the heredoc and Pi colors it like normal code.
• Smarter highlighting = the package keeps the same source text and layout, but adds stronger emphasis to important symbols in the normal bash tool row.
• Inspection backend = the package has a language-aware backend that can inspect the extracted snippet, even if the final tool row still uses basic highlighting.

Canonical package-backed capability tables

The package-backed capability tables now live in the host/package repo:

• Shipped today
• Researched next candidates

Use pi-inline-format-extensions as the source of truth for:

• which heredoc languages are currently shipped,
• which languages have real or partial inspection backends,
• and which next-language candidates were researched but are not wired yet.

This repo keeps the Rust-core and repo-local diagnostics documentation, while the package-backed capability matrix is maintained in Banon-Labs/pi-inline-format-extensions.

This repo still exposes extensions/index.ts through package.json because the repo-local Rust diagnostics remain a valid direct local entrypoint.

For normal package-backed development, project-scoped .pi/settings.json now loads both:

• host package source: git:github.com/Banon-Labs/pi-inline-format-extensions@v0.1.5
• local diagnostics extension: ../extensions/index.ts

This split keeps reusable runtime behavior in the pinned git-backed host package while preserving repo-local Rust CLI diagnostics inside pi-inline-format.

The intended stable release-time source is the pinned git ref above, resolved through the root-level Pi package surface in Banon-Labs/pi-inline-format-extensions.

For unpublished host changes under active development, the preferred local package source is now the repo root:

• ../../pi-inline-format-extensions

That root-level local path matches the same package surface used by future pinned git installs. The older ../../pi-inline-format-extensions/packages/host path remains relevant only as a historical migration source validated by npm run verify:host-source-upgrade-path.

Quick install in another project

If you are a normal user and just want the feature, install only the published npm package:

• npm:@banon-labs/pi-inline-format-extensions@0.1.5

If you need an exact repository build instead of the published package, the pinned git source remains available as an advanced fallback.

If that package is not in .pi/settings.json, this feature will not load.

You do not need:

• ../extensions/index.ts
• pi-tmux
• pi-semaphore
• any other local Pi package from this workspace

Create .pi/settings.json with just the published npm package source:

{
  "packages": [
    {
      "source": "npm:@banon-labs/pi-inline-format-extensions@0.1.5",
      "skills": [],
      "prompts": [],
      "themes": []
    }
  ]
}

Then, in that project:

pi list
pi

Inside Pi, these are good quick checks:

• /inline-format-host-status
• /inline-format-run-deterministic-compare javascript

That should show:

• the package under Project packages in pi list,
• host status text from the installed package,
• and the normal inline JavaScript heredoc row with hello from js 42.

This install path was validated in a clean temp project with no repo-local diagnostics extension loaded. The repo-local development flow below still uses a pinned git ref because current consumer-regression scripts are anchored to an exact host commit.

ANSI example: Python before and after install

This is the same Python heredoc prompt in two states:

• Before: deterministic provider only, no package-installed bash override.
• After: pinned package installed in .pi/settings.json.

The source text stays the same. The difference is that the installed package gives the Python body real syntax colors instead of one plain tool-colored block.

Before

\x1b[1m\x1b[38;2;208;144;96mdef main() -> None:\x1b[0m
\x1b[1m\x1b[38;2;208;144;96m    print("hello from /tmp/delete.me.py")\x1b[0m

After

\x1b[38;2;86;156;214mdef\x1b[39m \x1b[38;2;220;220;170mmain\x1b[39m() -> \x1b[38;2;181;206;168mNone\x1b[39m:
\x1b[38;2;78;201;176mprint\x1b[39m(\x1b[38;2;206;145;120m"hello from /tmp/delete.me.py"\x1b[39m)

These examples came from a clean install proof run captured in tmux.

Consumer install/update/migration flow

For normal users, prefer the npm package source above. The remaining pinned-git instructions in this repo are for repo-local development, release rehearsal, and exact-commit rollback.

Repo-local development flow

The section below is for this repo only.

If you are developing inside pi-inline-format, keep the pinned package and the local diagnostics extension:

{
  "packages": [
    {
      "source": "git:github.com/Banon-Labs/pi-inline-format-extensions@v0.1.5",
      "skills": [],
      "prompts": [],
      "themes": []
    }
  ],
  "extensions": ["../extensions/index.ts"]
}

What stays local vs package-backed

• Keep the host/runtime package source pinned in .pi/settings.json.
• Keep ../extensions/index.ts loaded locally for repo-local Rust CLI diagnostics.
• Do not move the local diagnostics commands into the package source; the package owns runtime seams, while this repo keeps diagnostics.

1. Land and publish the host-side changes in pi-inline-format-extensions first.
2. Replace the packages[0].source value in .pi/settings.json with the new pinned git ref.
3. Keep extensions[0] set to ../extensions/index.ts.
4. Run pi list and confirm the new pinned source appears under Project packages.
5. Run npm run check.
6. If the host change expanded language support or deterministic proof behavior, rerun the relevant proof flow before calling the repin complete.

ANSI-preserving tmux proof helper

For color-sensitive proof that should survive beyond a live pane, use:

npm run smoke:ansi-capture-proof

Default behavior:

• launches a deterministic target Pi pane in this repo,
• runs /inline-format-run-deterministic-compare typescript,
• launches an observer Pi pane,
• extracts ANSI-rich proof lines into a replay pane, then captures that replay pane with ANSI-preserving tmux capture-pane -e,
• validates that the observer capture artifact still contains ANSI-highlighted TypeScript output,
• writes artifacts under /tmp/pi-inline-smoke-ansi-capture-*/.

Useful flags:

• --scenario python|javascript|typescript|bash
• --keep-open to leave the dedicated smoke session running for inspection

This helper is for proofing the capture bridge itself. It complements, rather than replaces, the existing deterministic grid smoke and raw PI_TUI_WRITE_LOG artifacts. The automated harness entrypoint is npm run check:ansi-capture-proof.

Pre-release local-root validation flow

Before publishing a new host ref, validate against the local root package surface:

1. point packages[0].source at ../../pi-inline-format-extensions,
2. keep ../extensions/index.ts unchanged,
3. run npm run verify:multilanguage-local-package-proof,
4. restore the pinned git source after validation.

This flow is specifically for unpublished host work. Stable consumer installs should continue to use pinned git refs, not an unpinned sibling checkout.

Migrating from the old local sibling path

Older local development used:

• ../../pi-inline-format-extensions/packages/host

The verified migration path is:

1. start from the old local-path package source above,
2. switch packages[0].source to the pinned git source,
3. keep ../extensions/index.ts unchanged,
4. verify with npm run verify:host-source-upgrade-path,
5. finish with npm run check.

npm run verify:host-source-upgrade-path rehearses the exact transition locally by temporarily switching .pi/settings.json from the old local-path source to the pinned git source, running pi list plus headless deterministic compare verification at each step, and restoring the original settings file automatically even on failure.

Rollback note

If a future update fails, restore .pi/settings.json to the last known-good package source and rerun:

• pi list
• npm run check