Skip to content

docs(method): five cool-ideas cards from the 2026-05-30 wrap-up#388

Open
flyingrobots wants to merge 1 commit into
mainfrom
docs/cool-ideas-batch
Open

docs(method): five cool-ideas cards from the 2026-05-30 wrap-up#388
flyingrobots wants to merge 1 commit into
mainfrom
docs/cool-ideas-batch

Conversation

@flyingrobots
Copy link
Copy Markdown
Owner

@flyingrobots flyingrobots commented May 31, 2026

Summary

Five cool-ideas backlog cards captured from the post-merge brainstorm authorized after the Echo 0025 Phase 2 pre-flight review. None of these block Phase 2 RED. All are recorded so the ideas survive the next context reset.

Card Lane Punch
`PLATFORM_claude-think-remember.md` cool-ideas Acknowledges `--remember` already exists; proposes `--since` / `--repo` / `--branch` filters + ritualizing it in CLAUDE.md as the canonical session-start restore.
`PLATFORM_test-timing-history.md` cool-ideas Per-test timing telemetry to `target/test-timing-history.jsonl`; `xtask slow-tests --top / --branch / --since / --regressed-since`. Defends PR #383's dev-loop speedup against silent re-decay.
`WESLEY_ir-structural-diff.md` cool-ideas `wesley diff ` classifier: breaking / potentially safe / drift. Would have caught the entire 0024-design append-only-enum doc bug at the IR level. Pre-push gate + PR review bot.
`METHOD_generated-playback.md` cool-ideas Phase 3 playback docs become generated artifacts derived from cycle goals + RED test names + acceptance + witness commands. Sign-off slots persist across regens in a sibling file.
`PLATFORM_xtask-leash-audit-stub.md` cool-ideas Minimum-viable leash audit: read frontmatter, print active leashes with per-symbol grep counts. No state machine yet. Argues for shipping early because Phase 2 GREEN is the moment the first leash should fire.

Each card carries acceptance criteria and cross-references its companions:

  • test-timing ↔ wesley-gen-test-loop-speedup
  • wesley-diff ↔ wesley-gen-emitter-duplication
  • leash-audit ↔ METHOD_leash-files + jedit/leash/jedit-session-port

Why this matters

The 2026-05-30 brainstorm produced more value than the merge work it bracketed; carding it locks that value into the backlog so it doesn't evaporate into Think entries that need archaeology to recover. The cards are short by design — proposal + rationale + acceptance criteria, no implementation.

Companion (in another repo)

`jedit/docs/method/backlog/cool-ideas/JEDIT_why-is-this-character-here-debugger.md` — the jedit-side cool-ideas card from the same brainstorm. Lands separately in a jedit PR.

Test plan

  • `scripts/verify-local.sh pre-commit` clean.
  • Markdownlint clean (5 files, 0 errors).
  • Reviewer: read each card, push back on any that don't match the actual intent, or just merge. None of these are urgent.

Summary by CodeRabbit

  • Documentation
    • Added planning documentation for future enhancements including automated playback generation, improved session memory capabilities, test performance regression tracking, and schema analysis tools.

Captures the post-merge backlog batch authorized by the user after
the Echo 0025 Phase 2 pre-flight review. None of these block
Phase 2 RED; all are recorded so the ideas do not evaporate.

PLATFORM_claude-think-remember.md
  Two-part: enhancement (--since / --repo / --branch filters on the
  existing --remember surface) + ritualization (replace --recent
  with --remember in CLAUDE.md session-start guidance). Acknowledges
  that --remember already exists and was discovered mid-session.

PLATFORM_test-timing-history.md
  Per-test timing telemetry: append to target/test-timing-history.jsonl
  on every test run; surface via `xtask slow-tests --top / --branch
  / --since / --regressed-since`. Defends the PR #383 dev-loop speedup
  against silent re-decay; the 96-second test should have been visible
  before it became rage fuel.

WESLEY_ir-structural-diff.md
  `wesley diff <old-ir> <new-ir>` classifier: breaking (enum reorder,
  field removal, codec_id change, op-id preimage change) vs
  potentially safe (additive new operation/type) vs drift (stale
  inline codec_id). Pre-push gate + PR review bot. Would have caught
  the entire 0024 design-doc append-only-enum bug at the IR level.

METHOD_generated-playback.md
  Phase 3 playback docs become generated artifacts derived from
  cycle goals + RED test names + acceptance criteria + witness
  commands. Sign-off slots persist in a sibling file across regens.
  Eliminates the single largest source of stale METHOD docs without
  removing the human sign-off step.

PLATFORM_xtask-leash-audit-stub.md
  Minimum-viable leash audit: read leash frontmatter, print active
  leashes with per-symbol grep counts against the target repo. No
  state-machine yet. Justifies shipping early because Phase 2 GREEN
  is precisely the moment the first leash should fire.

Each card has acceptance criteria. Each cross-references its
companions (test-timing → wesley-gen-speedup, wesley-diff →
wesley-gen-emitter-duplication, leash-audit → leash convention).
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 31, 2026

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds five design documents to the backlog describing proposed tooling and workflow features: automated playback generation from test metadata, enhanced claude-think --remember filtering, per-test timing history with regression detection, leash audit scanning, and IR-based schema migration classification.

Changes

Backlog Proposals

Layer / File(s) Summary
Generated playback automation
docs/method/backlog/cool-ideas/METHOD_generated-playback.md
Proposes Phase 3 playback generation from cycle goals, test names, and witness scaffolds, with persistent human sign-offs and an xtask method playback <cycle> command.
Leash audit MVP
docs/method/backlog/cool-ideas/PLATFORM_xtask-leash-audit-stub.md
Specifies xtask leash-audit to scan leash frontmatter and report active leashes with symbol grep counts; defers automatic state transitions and PR movement.
claude-think --remember enhancements
docs/method/backlog/cool-ideas/PLATFORM_claude-think-remember.md
Adds filtering flags (--since, --repo, --branch) to claude-think --remember and updates ~/.claude/CLAUDE.md session-start ritual to use --remember for branch/cycle restoration.
Test timing regression detection
docs/method/backlog/cool-ideas/PLATFORM_test-timing-history.md
Describes JSONL append-only timing log with per-test duration, status, and branch metadata; defines xtask slow-tests with --regressed-since flag for median-based regression queries.
IR structural diff classifier
docs/method/backlog/cool-ideas/WESLEY_ir-structural-diff.md
Proposes wesley diff to classify schema migrations into Breaking, Potentially safe/Additive, and Drift categories with specific enum/field/codec/op-id patterns; emits JSON for CI gating and human-readable summaries.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

tooling

Poem

Five blueprints laid on the backlog's shelf,
each tool awaits its engineering self—
playback that writes itself with pride,
regressions caught before they hide,
and schemas that speak what breaks or bends. 🛠️

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the PR: it adds five documentation cards to the cool-ideas backlog, captured from a specific wrap-up session. It is specific, concise, and directly reflects the changeset content.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/cool-ideas-batch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17ce49fb76

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +45 to +47
Capture happens via a `cargo test` wrapper or a libtest reporter
hook (Rust nightly has unstable JSON output; stable can wrap
`--format=json` parsing). On Node side (jedit), wrap the
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct stable libtest JSON guidance

On stable Rust, this implementation path is not available: rustc --test ... --help marks -Z unstable-options as nightly-only, and invoking a test binary with --format json on rustc 1.90 errors that JSON is only accepted on nightly with -Z unstable-options. If this card becomes the acceptance recipe, the cargo test timing telemetry will be built around a command the repo’s stable toolchain cannot run; please either require nightly/nextest or describe a stable capture path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/method/backlog/cool-ideas/METHOD_generated-playback.md`:
- Around line 83-90: Add a reproducibility acceptance criterion that requires
byte-stable output: update the checklist in METHOD_generated-playback.md (the
section describing the xtask method playback <cycle> command that produces
docs/method/playback/<cycle>/playback.md) to state that identical inputs must
produce byte-identical playback.md files and that the sign-off mapping must
remain unchanged; also describe the minimal verification step (re-run xtask
method playback <cycle> on identical inputs and compare bytes and signoff
mapping) and state that this determinism guarantee is required for METHOD Phase
3 references to the generator to be canonical.
- Around line 44-56: The generator for METHOD_generated-playback.md lacks a
deterministic ordering and stable question-id algorithm; update the generator to
(1) derive a stable question-id from an immutable source (preferably an explicit
scaffold id if present, otherwise a content-stable hash of the question scaffold
text + canonical normalization) and store that id with each question, (2)
canonicalize/sort output deterministically (e.g., sort by question-id, then
section title, then test name) so sections never reorder across regens, and (3)
persist sign-offs in the sibling playback-signoffs.md keyed by that stable
question-id so renames or test output changes don’t remap signatures. Also add a
short contract note to METHOD_generated-playback.md describing the id derivation
and the canonical sort order so consumers know the stability guarantees.

In `@docs/method/backlog/cool-ideas/PLATFORM_claude-think-remember.md`:
- Around line 21-23: The code block shows a mismatched CLI name ("think
--remember [--brief] [--limit=N] [query]") while the rest of the card uses
"claude-think --remember ..."; pick the canonical executable name used elsewhere
(change the snippet to "claude-think --remember [--brief] [--limit=N] [query]")
so the documented invocation is consistent with the rest of the document and
avoid stale references to "think".
- Around line 99-100: The acceptance criteria for the claude-think CLI flag
--since (used by the --remember command) is ambiguous; pick and document a
single canonical input format (e.g., ISO 8601 durations) or explicitly specify
exact precedence and normalization rules between ISO8601 and human-friendly
forms. Update the acceptance line for "claude-think --remember --since
<duration>" to state the chosen canonical format (or precise parsing precedence
and how inputs are normalized to canonical form) so tests can rely on
deterministic parsing and hashing for functions that consume --since.

In `@docs/method/backlog/cool-ideas/PLATFORM_test-timing-history.md`:
- Around line 45-47: Update the documentation line that suggests stable Rust can
parse libtest JSON by removing or correcting the claim that "`cargo test --
--format=json`" works on stable; instead state that libtest JSON output is
unstable and requires nightly unstable options (e.g., `-Z unstable-options`) or
a nightly toolchain, and reference "libtest" and "`--format=json`" so the doc at
the Capture paragraph is clearly corrected.

In `@docs/method/backlog/cool-ideas/PLATFORM_xtask-leash-audit-stub.md`:
- Around line 89-91: The acceptance criteria for `cargo xtask leash-audit` lacks
a deterministic output ordering; update the doc for the task so the CLI's
printed list of active leashes and per-symbol grep hit counts is canonically
ordered (e.g., sort leashes by the `scaffold` field and, within each leash, sort
symbols lexicographically) and state this requirement in the acceptance criteria
so CI and reviews see stable, repeatable output when reading
`docs/method/backlog/leash/*.md` frontmatter.
- Around line 102-104: The cross-repo reference
`jedit/docs/method/backlog/leash/jedit-session-port.md` in the card is stale
because it appears as an in-repo path; update the link to an explicit external
repository URL or clearly mark it as external (e.g., "external:
github.com/OWNER/jedit/..." or full https://... link) so readers in this
repository can follow it reliably; locate the string
`jedit/docs/method/backlog/leash/jedit-session-port.md` in
PLATFORM_xtask-leash-audit-stub.md and replace it with the canonical external
URL or an explicit "external" note and ensure link format matches other docs
links.

In `@docs/method/backlog/cool-ideas/WESLEY_ir-structural-diff.md`:
- Around line 89-91: The pre-push gate incorrectly calls "wesley diff
origin/main HEAD" (passing git refs) while wesley diff expects two IR artifact
paths; update the pre-push hook (the script that gates pushes for changes to
.ir.json or .graphql) to first resolve or generate the two concrete IR JSON file
paths for the base and head (e.g., produce base.ir.json and head.ir.json from
origin/main and HEAD) and then invoke "wesley diff <base-ir.json>
<head-ir.json>"; ensure the hook fails the push when wesley reports breaking
changes and accepts non-breaking changes or a version bump.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c94568bf-0d21-4516-9628-235a5141af24

📥 Commits

Reviewing files that changed from the base of the PR and between 8017ccc and 17ce49f.

📒 Files selected for processing (5)
  • docs/method/backlog/cool-ideas/METHOD_generated-playback.md
  • docs/method/backlog/cool-ideas/PLATFORM_claude-think-remember.md
  • docs/method/backlog/cool-ideas/PLATFORM_test-timing-history.md
  • docs/method/backlog/cool-ideas/PLATFORM_xtask-leash-audit-stub.md
  • docs/method/backlog/cool-ideas/WESLEY_ir-structural-diff.md

Comment on lines +44 to +56
A markdown document where each playback question is a section, and
each section auto-fills:

- The question (from the scaffold).
- The matching RED test names and their pass/fail at last run.
- The witness command(s) and their last-recorded output.
- A boxed sign-off slot ("Sponsor 1: **_ / Sponsor 2: _**").

The doc is regenerable. If a test renames, the next regen picks up
the new name; if a witness command emits new output, the doc
reflects it. Sign-off slots persist across regens (they live in a
sibling `<cycle>/playback-signoffs.md` keyed by question id, so the
generated doc can rewrite freely without nuking signatures).
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Specify canonical ordering and stable question-id derivation.

The generator contract is underspecified for determinism. Without explicit sort rules and a stable question-id source, regen can reorder sections or remap sign-offs across runs/renames.

Proposed doc patch
 A markdown document where each playback question is a section, and
 each section auto-fills:
@@
 - The witness command(s) and their last-recorded output.
 - A boxed sign-off slot ("Sponsor 1: **_ / Sponsor 2: _**").
+
+Determinism contract:
+- Section order is canonical (sorted by explicit `question_id` from the scaffold).
+- Test names within a section are canonical (lexicographic ascending).
+- Witness commands within a section are canonical (scaffold declaration order).
+- `question_id` is an explicit immutable field in the scaffold (never derived from
+  mutable prose text), and sign-offs are keyed only by that id.

As per coding guidelines, “Documentation accuracy matters — especially anything touching determinism guarantees, hash stability, or canonical ordering.”

🧰 Tools
🪛 LanguageTool

[uncategorized] ~44-~44: Did you mean the formatting language “Markdown” (= proper noun)?
Context: ...hod/playback//playback.md ``` A markdown document where each playback question i...

(MARKDOWN_NNP)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/METHOD_generated-playback.md` around lines 44
- 56, The generator for METHOD_generated-playback.md lacks a deterministic
ordering and stable question-id algorithm; update the generator to (1) derive a
stable question-id from an immutable source (preferably an explicit scaffold id
if present, otherwise a content-stable hash of the question scaffold text +
canonical normalization) and store that id with each question, (2)
canonicalize/sort output deterministically (e.g., sort by question-id, then
section title, then test name) so sections never reorder across regens, and (3)
persist sign-offs in the sibling playback-signoffs.md keyed by that stable
question-id so renames or test output changes don’t remap signatures. Also add a
short contract note to METHOD_generated-playback.md describing the id derivation
and the canonical sort order so consumers know the stability guarantees.

Comment on lines +83 to +90
1. An `xtask method playback <cycle>` command exists and produces
`docs/method/playback/<cycle>/playback.md` from the inputs
above.
2. Re-running the command after a test rename produces a doc with
the new name and preserves prior sign-offs.
3. METHOD Phase 3 documentation references the generator as the
canonical path (hand-written playback docs become an exception,
not the default).
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Add a reproducibility acceptance criterion for byte-stable output.

Acceptance currently checks functional behavior but not determinism. Add a criterion that identical inputs produce byte-identical playback.md and unchanged signoff mapping.

Proposed doc patch
 3. METHOD Phase 3 documentation references the generator as the
    canonical path (hand-written playback docs become an exception,
    not the default).
+4. Two consecutive runs with identical inputs produce byte-identical
+   `docs/method/playback/<cycle>/playback.md`, and sign-off association
+   remains unchanged.

As per coding guidelines, “Documentation accuracy matters — especially anything touching determinism guarantees, hash stability, or canonical ordering.”

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
1. An `xtask method playback <cycle>` command exists and produces
`docs/method/playback/<cycle>/playback.md` from the inputs
above.
2. Re-running the command after a test rename produces a doc with
the new name and preserves prior sign-offs.
3. METHOD Phase 3 documentation references the generator as the
canonical path (hand-written playback docs become an exception,
not the default).
1. An `xtask method playback <cycle>` command exists and produces
`docs/method/playback/<cycle>/playback.md` from the inputs
above.
2. Re-running the command after a test rename produces a doc with
the new name and preserves prior sign-offs.
3. METHOD Phase 3 documentation references the generator as the
canonical path (hand-written playback docs become an exception,
not the default).
4. Two consecutive runs with identical inputs produce byte-identical
`docs/method/playback/<cycle>/playback.md`, and sign-off association
remains unchanged.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/METHOD_generated-playback.md` around lines 83
- 90, Add a reproducibility acceptance criterion that requires byte-stable
output: update the checklist in METHOD_generated-playback.md (the section
describing the xtask method playback <cycle> command that produces
docs/method/playback/<cycle>/playback.md) to state that identical inputs must
produce byte-identical playback.md files and that the sign-off mapping must
remain unchanged; also describe the minimal verification step (re-run xtask
method playback <cycle> on identical inputs and compare bytes and signoff
mapping) and state that this determinism guarantee is required for METHOD Phase
3 references to the generator to be canonical.

Comment on lines +21 to +23
```text
think --remember [--brief] [--limit=N] [query]
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix CLI name mismatch in the declared current surface.

This block documents think --remember ..., but the rest of the card specifies claude-think --remember .... Keep one canonical executable name to avoid stale/incorrect invocation docs.

Proposed diff
- think --remember [--brief] [--limit=N] [query]
+ claude-think --remember [--brief] [--limit=N] [query]

As per coding guidelines, "Documentation accuracy matters — especially anything touching determinism guarantees, hash stability, or canonical ordering. Flag factual errors and stale cross-references."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/PLATFORM_claude-think-remember.md` around
lines 21 - 23, The code block shows a mismatched CLI name ("think --remember
[--brief] [--limit=N] [query]") while the rest of the card uses "claude-think
--remember ..."; pick the canonical executable name used elsewhere (change the
snippet to "claude-think --remember [--brief] [--limit=N] [query]") so the
documented invocation is consistent with the rest of the document and avoid
stale references to "think".

Comment on lines +99 to +100
1. `claude-think --remember --since <duration>` accepts ISO 8601
duration or human-friendly forms and filters correctly.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make --since duration format canonical in acceptance criteria.

“ISO 8601 duration or human-friendly forms” defines two grammars for one flag and leaves parsing precedence unspecified. For deterministic behavior and reproducible tests, specify a single canonical input format (or define exact precedence + normalization rules).

Proposed diff
-1. `claude-think --remember --since <duration>` accepts ISO 8601
-   duration or human-friendly forms and filters correctly.
+1. `claude-think --remember --since <duration>` accepts a single
+   canonical duration format (ISO 8601) and filters correctly.

As per coding guidelines, "Documentation accuracy matters — especially anything touching determinism guarantees, hash stability, or canonical ordering."

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
1. `claude-think --remember --since <duration>` accepts ISO 8601
duration or human-friendly forms and filters correctly.
1. `claude-think --remember --since <duration>` accepts a single
canonical duration format (ISO 8601) and filters correctly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/PLATFORM_claude-think-remember.md` around
lines 99 - 100, The acceptance criteria for the claude-think CLI flag --since
(used by the --remember command) is ambiguous; pick and document a single
canonical input format (e.g., ISO 8601 durations) or explicitly specify exact
precedence and normalization rules between ISO8601 and human-friendly forms.
Update the acceptance line for "claude-think --remember --since <duration>" to
state the chosen canonical format (or precise parsing precedence and how inputs
are normalized to canonical form) so tests can rely on deterministic parsing and
hashing for functions that consume --since.

Comment on lines +45 to +47
Capture happens via a `cargo test` wrapper or a libtest reporter
hook (Rust nightly has unstable JSON output; stable can wrap
`--format=json` parsing). On Node side (jedit), wrap the
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

On current Rust stable, is cargo test -- --format=json officially supported without unstable flags? Please provide rustc book/libtest docs links.

💡 Result:

Not on current Rust stable: cargo test -- --format=json relies on libtest’s --format json option, which is documented as unstable and requires -Z unstable-options (i.e., an unstable flag) to use [1]. Relevant docs (Rust Book / libtest): 1) The rustc book “Tests” chapter documents --format FORMAT and states that json “is unstable, and requires the -Z unstable-options flag” [1]. Link: https://doc.rust-lang.org/rustc/tests/ 2) The rustc book “JSON Output” chapter documents rustc’s JSON error format flags (different from libtest’s --format json). It notes its JSON output is currently unstable and requires -Z unstable-options [2]. Link: https://doc.rust-lang.org/rustc/json.html So, for machine-readable test output from cargo test, you generally need unstable flags for the upstream libtest JSON option, unless you use external tooling that supports an experimental/stabilized message format on its side [3].

Citations:


Fix incorrect Rust stable JSON test-output guidance

cargo test -- --format=json relies on libtest’s JSON output, which is documented as unstable and requires -Z unstable-options (Rust “Tests” chapter: https://doc.rust-lang.org/rustc/tests/). Remove the suggestion that stable can parse it via --format=json.

Proposed wording fix
-Capture happens via a `cargo test` wrapper or a libtest reporter
-hook (Rust nightly has unstable JSON output; stable can wrap
-`--format=json` parsing). On Node side (jedit), wrap the
+Capture happens via a `cargo test` wrapper or a libtest reporter
+hook (structured libtest JSON output is a nightly/unstable path;
+on stable, use a wrapper strategy that records per-test timings
+from supported output/reporting surfaces). On Node side (jedit), wrap the
 `node --test` JSON reporter.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Capture happens via a `cargo test` wrapper or a libtest reporter
hook (Rust nightly has unstable JSON output; stable can wrap
`--format=json` parsing). On Node side (jedit), wrap the
Capture happens via a `cargo test` wrapper or a libtest reporter
hook (structured libtest JSON output is a nightly/unstable path;
on stable, use a wrapper strategy that records per-test timings
from supported output/reporting surfaces). On Node side (jedit), wrap the
`node --test` JSON reporter.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/PLATFORM_test-timing-history.md` around lines
45 - 47, Update the documentation line that suggests stable Rust can parse
libtest JSON by removing or correcting the claim that "`cargo test --
--format=json`" works on stable; instead state that libtest JSON output is
unstable and requires nightly unstable options (e.g., `-Z unstable-options`) or
a nightly toolchain, and reference "libtest" and "`--format=json`" so the doc at
the Capture paragraph is clearly corrected.

Comment on lines +89 to +91
1. `cargo xtask leash-audit` reads every
`docs/method/backlog/leash/*.md` frontmatter and prints the
active leashes with per-symbol grep hit counts.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Specify canonical output ordering in acceptance criteria.

The acceptance criteria requires printing active leashes and symbol hit counts but does not define ordering. For CI/review reproducibility, require a stable order (e.g., sort leashes by scaffold and symbols lexicographically) so output is deterministic across runs.

As per coding guidelines, "Documentation accuracy matters — especially anything touching determinism guarantees, hash stability, or canonical ordering."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/PLATFORM_xtask-leash-audit-stub.md` around
lines 89 - 91, The acceptance criteria for `cargo xtask leash-audit` lacks a
deterministic output ordering; update the doc for the task so the CLI's printed
list of active leashes and per-symbol grep hit counts is canonically ordered
(e.g., sort leashes by the `scaffold` field and, within each leash, sort symbols
lexicographically) and state this requirement in the acceptance criteria so CI
and reviews see stable, repeatable output when reading
`docs/method/backlog/leash/*.md` frontmatter.

Comment on lines +102 to +104
- `jedit/docs/method/backlog/leash/jedit-session-port.md` — the
first leash, and the one that will make this tool
immediately useful.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Companion link is a stale cross-repo reference as written.

jedit/docs/... is presented like an in-repo path, but this card is in a different repository context. Use an explicit external repo URL (or clearly mark it as external) to avoid a broken/stale cross-reference.

As per coding guidelines, "Documentation accuracy matters — especially anything touching determinism guarantees, hash stability, or canonical ordering. Flag factual errors and stale cross-references."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/PLATFORM_xtask-leash-audit-stub.md` around
lines 102 - 104, The cross-repo reference
`jedit/docs/method/backlog/leash/jedit-session-port.md` in the card is stale
because it appears as an in-repo path; update the link to an explicit external
repository URL or clearly mark it as external (e.g., "external:
github.com/OWNER/jedit/..." or full https://... link) so readers in this
repository can follow it reliably; locate the string
`jedit/docs/method/backlog/leash/jedit-session-port.md` in
PLATFORM_xtask-leash-audit-stub.md and replace it with the canonical external
URL or an explicit "external" note and ensure link format matches other docs
links.

Comment on lines +89 to +91
- Pre-push gate on any change that touches a `.ir.json` or a
`.graphql` schema: run `wesley diff origin/main HEAD` and refuse
the push if breaking changes are present without a version bump.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fix pre-push invocation to pass IR artifacts, not git refs.

wesley diff is specified as wesley diff <old-ir> <new-ir>, but wesley diff origin/main HEAD passes commit refs, not IR files. This makes the documented gate command incorrect and non-runnable unless hidden behavior exists.

Use an invocation that resolves/generates two concrete IR JSON paths (base vs head) before calling wesley diff.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/method/backlog/cool-ideas/WESLEY_ir-structural-diff.md` around lines 89
- 91, The pre-push gate incorrectly calls "wesley diff origin/main HEAD"
(passing git refs) while wesley diff expects two IR artifact paths; update the
pre-push hook (the script that gates pushes for changes to .ir.json or .graphql)
to first resolve or generate the two concrete IR JSON file paths for the base
and head (e.g., produce base.ir.json and head.ir.json from origin/main and HEAD)
and then invoke "wesley diff <base-ir.json> <head-ir.json>"; ensure the hook
fails the push when wesley reports breaking changes and accepts non-breaking
changes or a version bump.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant