Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -832,7 +832,9 @@ acorn / Svelte / `parseCss` over the author's own bytes. **Every parse-then-form
point folds `<CR>` / `<CR><LF>` to `<LF>` before it parses**
(`tsv_lang::printing::normalize_carriage_returns` — each language crate's `format_str`, the
CLI's `format_source`, each binding's format export, `canonicalize_js`), so tsv's output is
LF-only even inside the regions it copies verbatim. Ahead of the parse is the only place
LF-only even inside the regions it copies verbatim. The fold's one pass also takes the
folded document's line verdict (`FoldedSource`), which each crate's `format_folded_in`
hands its printer, so a document that folds is walked once. Ahead of the parse is the only place
that answers it once: the printers ask "where are the lines?" in several places that split
on `'\n'` alone, and folding the finished string instead leaves those disagreeing with the
output — the same document then formats two ways on two passes. `<LS>` / `<PS>` are
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ tsv/
Each language crate exports a consistent API:

- `parse(source, arena) -> Result<AST>` — the AST allocates into the caller's `bumpalo` arena (the bindings reuse a per-thread arena across calls via `tsv_arena`)
- `format(ast, source) -> String` — plus `format_in(ast, source, doc_arena)`, the same formatter writing through a reusable doc arena for the bindings' hot loop, and `format_str(source)`, the parse+format one-shot
- `format(ast, source) -> String` — plus `format_in(ast, source, doc_arena)`, the same formatter writing through a reusable doc arena for the bindings' hot loop, `format_folded_in(ast, &folded, doc_arena)` for a caller that folded the source's line terminators itself, and `format_str(source)`, the parse+format one-shot
- `convert_ast_json_bytes(ast, source) -> Vec<u8>` — the wire JSON, emitted directly from the internal AST, with `convert_ast_json_string`/`convert_ast_json` wrappers and span-only `_no_locations` variants alongside (default-on `convert` cargo feature; turn off for parse+format-only builds)

For more details see [CLAUDE.md](CLAUDE.md).
Expand Down
14 changes: 8 additions & 6 deletions crates/tsv_cli/src/cli/format_source.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,18 +68,20 @@ pub fn format_source_in_with_goal(
// printer never sees a `<CR>` and the doc-build's line splits agree with the output
// about where the lines are. See `tsv_lang::printing::normalize_carriage_returns`; the
// `parse` command deliberately skips it, its offsets being a drop-in contract over the
// author's own bytes. Borrowed unchanged on a source with no `<CR>`.
let source = tsv_lang::printing::normalize_carriage_returns(source);
let source = source.as_ref();
// author's own bytes. Borrowed unchanged on a source with no `<CR>`, and the fold's
// pass is the document's line verdict too — the `format_folded_in` siblings take it
// rather than walking the source again.
let folded = tsv_lang::printing::normalize_carriage_returns(source);
let source = folded.text();
match parser_type {
ParserType::Svelte => tsv_svelte::parse(source, arena)
.map(|ast| tsv_svelte::format_in(&ast, source, doc_arena))
.map(|ast| tsv_svelte::format_folded_in(&ast, &folded, doc_arena))
.map_err(|e| e.to_string()),
ParserType::Css => tsv_css::parse(source, arena)
.map(|ast| tsv_css::format_in(&ast, source, doc_arena))
.map(|ast| tsv_css::format_folded_in(&ast, &folded, doc_arena))
.map_err(|e| e.to_string()),
ParserType::TypeScript => tsv_ts::parse_with_goal(source, goal, arena)
.map(|ast| tsv_ts::format_in(&ast, source, doc_arena))
.map(|ast| tsv_ts::format_folded_in(&ast, &folded, doc_arena))
.map_err(|e| e.to_string()),
}
}
4 changes: 2 additions & 2 deletions crates/tsv_css/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Standard `ast/lexer/parser/printer` crate layout — see [root CLAUDE.md §Proje
**Standalone** (top-level CSS files):

- `parse(source, arena: &'arena Bump) -> Result<CssStyleSheet<'arena>>` — parse a full CSS file. The internal AST is bump-arena-allocated (caller-owns-`Bump`); `CssStyleSheet<'arena>` borrows from it. Matches `tsv_ts`/`tsv_svelte`'s signature for the shared `lang_bindings!` macro + CLI/FFI/WASM callers.
- `format_str(source) -> Result<String>` — parse + `format` fused (applies the format path's `<CR>` fold ahead of the parse), the same twin every language crate has
- `format(&stylesheet, source) -> String` — format with default config; `format_in(&stylesheet, source, &DocArena)` is the same into a caller-provided doc arena (multi-file drivers reuse one via `DocArena::reset()`; `format` is the fresh-arena wrapper)
- `format_str(source) -> Result<String>` — parse + `format` fused (applies the format path's `<CR>` fold ahead of the parse and formats through `format_folded_in`), the same twin every language crate has
- `format(&stylesheet, source) -> String` — format with default config; `format_in(&stylesheet, source, &DocArena)` is the same into a caller-provided doc arena (multi-file drivers reuse one via `DocArena::reset()`; `format` is the fresh-arena wrapper); `format_folded_in(&stylesheet, &FoldedSource, &DocArena)` is `format_in` for a caller that folded the source itself — identical output, the line verdict taken from the fold's pass (what the CLI, the bindings and `format_str` call)
- `convert_ast_json_bytes(&stylesheet, source) -> Vec<u8>` / `convert_ast_json_string(...) -> String` — the **sole emission path** (bytes: FFI/CLI non-pretty; string: the same bytes plus one output UTF-8 validation, for WASM `JSON.parse` / N-API): the writer (`ast/convert/write.rs`) walks the *internal* AST once and emits the wire JSON directly into a pre-sized buffer (`tsv_lang::estimated_json_capacity`), never materializing a typed public tree, fusing byte→char offset translation into the walk (each position through a `ByteToCharMap`; identity on ASCII). Matches Svelte's `parseCss()` JSON shape, reusing the raw-source reconstruction helpers in `ast/convert/mod.rs` (`strip_css_comments`, `split_declaration_svelte_compat`, `raw_selector_name`, …). Gated against the canonical `parseCss` `expected.json` and `corpus:compare:parse --multibyte-only`
- `convert_ast_json(&stylesheet, source) -> serde_json::Value` — a thin wrapper (`serde_json::from_slice(&convert_ast_json_bytes(...))`) for the `Value` consumers (the CLI's `--pretty`, the fixture gate); not an independent conversion
- `convert_ast_json_bytes_no_locations(...)` / `convert_ast_json_string_no_locations(...)` — the span-only variant, present only for parity with the TS/Svelte writers and the uniform `lang_bindings!` macro. `parseCss` emits **no** per-node `loc`, so the CSS wire is already offset-only — these are **exact aliases** of the plain functions (a documented no-op, not a distinct shape). The CLI/FFI/N-API/WASM `no-locations` surfaces therefore treat the option as inert for CSS — their generated CSS rows exist for macro uniformity and are byte-identical to the plain parse.
Expand Down
19 changes: 16 additions & 3 deletions crates/tsv_css/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,11 @@ pub fn format_str(source: &str) -> Result<String> {
// The format path's line-terminator fold, ahead of the parse (see
// `tsv_lang::printing::normalize_carriage_returns`); `parse` leaves the author's bytes
// alone so its offsets stay a drop-in contract with `parseCss`'s.
let source = tsv_lang::printing::normalize_carriage_returns(source);
let folded = tsv_lang::printing::normalize_carriage_returns(source);
let arena = bumpalo::Bump::new();
let stylesheet = parse(&source, &arena)?;
Ok(format(&stylesheet, &source))
let stylesheet = parse(folded.text(), &arena)?;
let doc_arena = tsv_lang::doc::arena::DocArena::for_source(folded.text());
Ok(format_folded_in(&stylesheet, &folded, &doc_arena))
}

/// Format into a caller-provided doc arena.
Expand All @@ -119,6 +120,18 @@ pub fn format_in(
printer::format_css_in(stylesheet, source, arena)
}

/// [`format_in`] over a document the caller folded ahead of the parse
/// (`tsv_lang::printing::normalize_carriage_returns`) — the format entry points that fold
/// (the CLI, the bindings, [`format_str`]). Identical output; the document's line verdict
/// comes from the fold's own pass instead of a second walk of the source.
pub fn format_folded_in(
stylesheet: &CssStyleSheet<'_>,
folded: &tsv_lang::printing::FoldedSource<'_>,
arena: &tsv_lang::doc::arena::DocArena,
) -> String {
printer::format_css_folded_in(stylesheet, folded, arena)
}

/// Format an embedded CSS stylesheet into a caller-provided doc arena.
///
/// `embed.base_indent_offset` seeds the indent so wrapped lines respect the host's
Expand Down
32 changes: 27 additions & 5 deletions crates/tsv_css/src/printer/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ use tsv_lang::{
arena::{DocArena, DocId},
},
is_format_ignore_directive,
printing::{self, LineBreaks, LineTable},
printing::{self, FoldedSource, LineBreaks, LineTable},
};

/// Printer state for building output
Expand Down Expand Up @@ -1150,17 +1150,39 @@ pub(crate) fn format_css_in(
stylesheet: &CssStyleSheet<'_>,
source: &str,
arena: &DocArena,
) -> String {
// The document's line verdict, over the arena-parked table (one warm table across a
// multi-file driver's files instead of a fresh Vec per file — filled only if a line
// question falls back to it).
let line_breaks = LineBreaks::new(source, arena.take_line_breaks_scratch());
format_stylesheet(stylesheet, source, line_breaks, arena)
}

/// [`format_css_in`] over a document the caller folded ahead of the parse: the line
/// verdict comes from the fold's own pass (`LineBreaks::of_folded`), not a second walk.
pub(crate) fn format_css_folded_in(
stylesheet: &CssStyleSheet<'_>,
folded: &FoldedSource<'_>,
arena: &DocArena,
) -> String {
let line_breaks = LineBreaks::of_folded(folded, arena.take_line_breaks_scratch());
format_stylesheet(stylesheet, folded.text(), line_breaks, arena)
}

/// The shared body of the two: register the stylesheet's comments, build the printer on
/// its line table, print, park the table.
fn format_stylesheet(
stylesheet: &CssStyleSheet<'_>,
source: &str,
line_breaks: LineBreaks<'_>,
arena: &DocArena,
) -> String {
// The print-once comment ledger's expectation for this stylesheet — detached comments
// plus in-block `CssBlockChild::Comment` AST nodes (diagnostic; see
// `tsv_lang::comment_ledger`).
#[cfg(feature = "comment_check")]
register_stylesheet_comments(stylesheet, source);

// The document's line verdict, over the arena-parked table (one warm table across a
// multi-file driver's files instead of a fresh Vec per file — filled only if a line
// question falls back to it).
let line_breaks = LineBreaks::new(source, arena.take_line_breaks_scratch());
let mut printer = Printer::new(arena, source, &stylesheet.comments, line_breaks.table());
printer.print_css_nodes(stylesheet.nodes);
let output = printer.into_string();
Expand Down
2 changes: 1 addition & 1 deletion crates/tsv_debug/src/audit/census.rs
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ pub(crate) fn comment_census(source: &str, parser: ParserType) -> CensusMultiset
/// make `a<LS>b` and `a<LF>b` compare equal and blind the census to a real rewrite. `\r`
/// left the trim class in the same step — after the fold there is none left to trim.
fn normalize_interior(kind: CensusKind, raw: &str) -> String {
let raw = tsv_lang::printing::normalize_carriage_returns(raw);
let raw = tsv_lang::printing::normalize_carriage_returns(raw).into_text();
match kind {
// Runs to end of line, so it has exactly one edge the printer touches.
CensusKind::Line => raw.trim_end_matches(tsv_lang::is_js_whitespace).to_owned(),
Expand Down
5 changes: 0 additions & 5 deletions crates/tsv_debug/src/cli/commands/scan_audit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -142,11 +142,6 @@ const ALLOW: &[Allow] = &[
"if let Some(last_newline_pos) = s.rfind('\\n') {",
"non-source",
),
(
"tsv_lang/src/printing.rs",
"let Some(first) = source.find('\\r') else {",
"terminator-fold",
),
(
"tsv_lang/src/printing.rs",
"while let Some(i) = rest.find('\\r') {",
Expand Down
2 changes: 1 addition & 1 deletion crates/tsv_ffi/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Depends on `tsv_ts`, `tsv_css`, `tsv_svelte`. Sibling binding crates: [`tsv_wasm`](../tsv_wasm/) (WebAssembly) and [`tsv_napi`](../tsv_napi/) (N-API, the Node/Bun native path). This crate is the C-ABI path — consumers include Deno FFI, Python `ctypes`, and any other C-FFI host. Node/Bun use `tsv_napi` instead (no C-FFI glue); the per-thread arena reuse below is shared across all three bindings via the [`tsv_arena`](../tsv_arena/) crate.

The bindings reuse a **per-thread AST `Bump`** (`with_ast_arena`) that is `reset()` between calls rather than allocated fresh per call: the bindings are invoked once per file in tight loops, and per-call arena malloc/free churns the system allocator's heap high-water in a way that is measurable through a host FFI layer. `reset()` retains the largest chunk and rewinds, so a warm thread does no per-call malloc/free. The per-file AST is fully consumed before the next call's `reset()`, so the reuse is sound (incl. after a `catch_unwind`-caught panic). The `format` path additionally reuses a **per-thread doc arena** (`with_doc_arena`, the same shape over `DocArena` and calling each language's `format_in`). Both helpers live in the shared [`tsv_arena`](../tsv_arena/) crate (one copy for all three bindings — `tsv_ffi`, `tsv_napi`, `tsv_wasm`); this crate's `format` feature maps to `tsv_arena/format`, which pulls `tsv_lang` for the `DocArena` type, so the parse-only build stays lean.
The bindings reuse a **per-thread AST `Bump`** (`with_ast_arena`) that is `reset()` between calls rather than allocated fresh per call: the bindings are invoked once per file in tight loops, and per-call arena malloc/free churns the system allocator's heap high-water in a way that is measurable through a host FFI layer. `reset()` retains the largest chunk and rewinds, so a warm thread does no per-call malloc/free. The per-file AST is fully consumed before the next call's `reset()`, so the reuse is sound (incl. after a `catch_unwind`-caught panic). The `format` path additionally reuses a **per-thread doc arena** (`with_doc_arena`, the same shape over `DocArena` and calling each language's `format_folded_in` — the fold ahead of the parse hands its line verdict to the printer, so a document is walked once). Both helpers live in the shared [`tsv_arena`](../tsv_arena/) crate (one copy for all three bindings — `tsv_ffi`, `tsv_napi`, `tsv_wasm`); this crate's `format` feature maps to `tsv_arena/format`, which pulls `tsv_lang` for the `DocArena` type, so the parse-only build stays lean.

Build/usage commands live in [../../CLAUDE.md §JS Bindings](../../CLAUDE.md#js-bindings).

Expand Down
6 changes: 3 additions & 3 deletions crates/tsv_ffi/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -268,13 +268,13 @@ macro_rules! parse_format {
// The format path's line-terminator fold, ahead of the parse — see
// `tsv_lang::printing::normalize_carriage_returns`. `parse_convert!` deliberately
// skips it: the wire's offsets are a drop-in contract over the author's own bytes.
let normalized = tsv_lang::printing::normalize_carriage_returns($source);
let source = normalized.as_ref();
let folded = tsv_lang::printing::normalize_carriage_returns($source);
let source = folded.text();
with_ast_arena(|arena| {
let ast =
parse_ast!($goalness, $lang, source, $goal, arena).map_err(|e| e.to_string())?;
Ok(with_doc_arena(|doc_arena| {
$lang::format_in(&ast, source, doc_arena)
$lang::format_folded_in(&ast, &folded, doc_arena)
}))
})
}};
Expand Down
Loading