v0.3.1

v0.3.1

Versioned as a patch despite the breaking changes below: they touch only the block-type
and sentence-splitting surface introduced in v0.3.0, which is very new and not yet relied
upon. (The pre-1.0 policy otherwise bumps the minor version for breaking changes.)

Makes TextDoc block-aware end to end and adds the DocGraph node model on top of it.
The block-aware layer gives an exact-span structural block tree, a section hierarchy
with rolled-up stats, inline-link rollups, and link-aware sentence spans. The DocGraph
layer then adds a recursive, layer-tagged node table, the base_blocks() sequential
partition, the collect() query primitive, the SpanRef span-reference type, and the
DocGraph Pydantic projection (schema “DocGraph/v0.1”).
The source text and its offset space are canonical; every view (the structural block
tree, sections, block-type slices, tallies, and the node table) is a projection over
that substrate: no stored counts.
Block boundaries and spans now come straight from flowmark’s parser, so chopdiff carries
no Markdown block-detection regex of its own.

Breaking Changes

• BlockType.list is now bullet-only; ordered lists are BlockType.ordered_list.
  Ordered-ness is carried from marko’s List.ordered. Callers that matched
  BlockType.list to cover both list kinds now miss ordered lists; match
  {BlockType.list, BlockType.ordered_list} for either.
• Default sentence splitting is now span-aware. TextDoc.from_text() with the
  default splitter now routes through flowmark.atomic_spans.split_sentences_with_spans
  instead of calling split_sentences_regex directly, so sentences never bisect a link,
  code span, or autolink.
  Sentence boundaries can therefore differ from v0.3.0 for text containing those
  constructs. default_sentence_splitter is unchanged and passing an explicit splitter
  preserves the previous behavior.

New Features

• Opt-in structural block tree with exact spans. TextDoc.blocks() returns a
  Block(type, span, children, tight) tree whose boundaries and [start, end) spans
  come directly from flowmark’s parser (marko’s own source positions), so a fenced code
  block stays whole through internal blank lines and a list always decomposes into
  list_items with nested sublists.
  The tree is density-invariant: tight and loose spacing of the same list produce
  identical block/item counts; Block.tight records the CommonMark spacing.
• Per-section structure and tallies. Section.blocks() scopes the structural tree
  to a section’s own content (document-absolute spans), and
  Section.block_type_counts() / TextDoc.block_type_counts() give derived
  Counter[BlockType] tallies over the live tree (no stored counts).
  Section.content holds the section’s own paragraphs (renamed from Section.blocks,
  which is now the structural method).
• Sections, TOC, and rolled-up size stats. TextDoc.sections() returns a tree of
  Sections over the heading hierarchy; TextDoc.toc() returns a flat
  (level, title, span) list.
  Section.size(unit, subtree=True|False), Section.size_summary(), and
  TextDoc.section_size_tree(units=…) roll up sizes per section in any TextUnit.
• Exact [start, end) spans on paragraphs and sentences. Every Paragraph and
  Sentence exposes a document-relative span; TextDoc.source_text is retained so
  each unit’s original_text round-trips into the source.
  TextDoc.block_at_offset(o) and sentence_at_offset(o) invert spans.
• Inline-link rollups and link-aware sentence spans. Link(text, url, title, span)
  via Paragraph.links(), Section.links(), and TextDoc.links()—identity from
  flowmark’s extract_links (reference links resolve across the whole document), spans
  recovered from iter_atomic_spans. The default sentence splitter is now
  flowmark.atomic_spans.split_sentences_with_spans, so sentence spans are exact for
  all content and never bisect a link, code span, or autolink.
• More block types: BlockType gains ordered_list, list_item, and
  thematic_break.
• New public exports: Block, Link, Section.
• Recursive node table with layers. The canonical node table fully populates
  container children (blockquotes, list items) and tags each node with its parse layer
  (textual, markdown, document, synthetic).
  Cross-layer relationships are offset-containment queries over a shared id space.
• base_blocks() sequential partition. A flat, depth-annotated, non-overlapping
  partition whose spans cover every non-whitespace character exactly once.
  Lists decompose so each list item is its own base block with increasing depth
  (list-item continuation content keeps its own real type, e.g. paragraph, not
  list_item); blockquotes stay atomic.
  Exact source reconstruction is via each block’s source_span (not by concatenating
  block text).
• collect() query primitive. One general query
  (collect(kinds=, where=, recursive=, inline=, layer=)) at document, section, and
  block scope, superseding block_type_counts() convenience accessors.
  The layer= filter scopes a query to one or more parse layers (default: all layers),
  since the same span can appear as nodes in several layers.
  Two relation families select candidates: the tree relation subtree_of= (a node's
  within-layer subtree) and the cross-layer interval relations within= /overlaps=
  (each takes a node id or a span), so within=section_id gathers everything inside a
  section without recursive=True. (scope=/contains= remain as deprecated aliases.)
• SpanRef span-reference type. Quote-canonical, offset-hinted span references for
  durable annotation anchoring (exact-quote resolution with prefix/suffix
  disambiguation; fuzzy re-anchoring deferred).
• DocGraph Pydantic projection. TextDoc.graph(include=, detail=) builds a
  serialized, language-neutral JSON contract (schema “DocGraph/v0.1”) with composable
  Layer and Detail axes.
• TextDoc.base_blocks() method. A thin method over the base_blocks free
  function, so the sequential partition has the same ergonomic surface as blocks().
• Reusable debug dumper (chopdiff.docs.debug). doc_report, doc_graph_yaml, and
  dump_views turn any document into clean, deterministic standard-format views (a
  multi-view report, the DocGraph, the reassembled source) for REPL/script debugging and
  golden testing.
• DocGraph.to_yaml(). A clean, deterministic YAML serialization of the projection
  alongside the existing JSON (block style, | block scalars, None/empty suppressed).

Fixes

• base_blocks() complete-cover fix. Content following (or between) a nested
  sublist inside a list item was dropped from the partition; the partition is now a
  complete cover again (verified by a cover-invariant test over all non-whitespace
  source).
• Structural-parse memoization. TextDoc.blocks() is now cached on the immutable
  source_text, and Section.blocks() / Section.links() slice that single cached
  parse instead of re-parsing the whole document per section (a TOC walk was quadratic).
• Sections built from structural headings. sections()/toc() now derive headings
  from the structural parse (top-level heading blocks) instead of the blank-line
  paragraph view, so a #-prefixed line isolated from inside a fenced code block is no
  longer mistaken for a section heading. Behavior change: such phantom sections no longer
  appear (e.g. the malformed golden loses one).
• Linear node-table assembly. Inline-element attribution (containing block, section,
  and sentence) now goes through a per-layer IntervalIndex instead of scanning the
  whole table per inline element, so build_node_table is linear rather than
  O(inline × nodes) on link-heavy or large documents.
• Single shared parse with thread-safe caching. blocks(), links(), and
  base_blocks() now derive from one cached marko parse of source_text instead of each
  re-parsing the whole document, roughly halving node_table() build time (one full
  parse instead of two). Document-level derivations are memoized under a per-instance
  reentrant lock, so concurrent reads compute each cache at most once and observe the
  same value; reads are otherwise side-effect-free and deterministic. See the TextDoc
  read-time-caching contract.

Internal

• Dropped chopdiff’s regex block scanner. TextDoc.blocks() and
  Paragraph.block_type now walk flowmark’s annotated parse tree and map marko classes
  to BlockType through a single table; the per-line regex scanner, classify_block,
  and the cached markdown_parser singleton are gone (net negative code).
  Because chopdiff no longer makes block-boundary decisions, two earlier bugs are fixed
  by construction: reference links resolve across block boundaries, and adjacent blocks
  with no blank line between them split correctly.

Documentation

• Grounded design principles. docs/textdoc-spec.md now leads with an explicit,
  three-tier principle set (P1–P18) and a pitfalls/decisions note; the goals cite the
  principles they realize.
  The canonical substrate is stated as the source text + offset space, with the node
  table as one projection (not a rival store).

Dependencies

• New runtime dependency: pydantic>=2.13.4 (brings annotated-types, pydantic-core,
  typing-inspection as transitive dependencies).
  Required for the DocGraph schema.
• New runtime dependency: frontmatter-format>=0.3.0 (first-party; brings
  ruamel-yaml). Used for clean deterministic YAML (DocGraph.to_yaml, the debug
  dumper) and the Markdown-with-frontmatter golden-test corpus.
• Requires flowmark>=0.7.1 for the authoritative block spans
  (jlevy/flowmark#52); recorded as a
  reviewed first-party cool-off exception in SUPPLY-CHAIN-SECURITY.md. No new
  transitive dependencies over 0.7.0.

Compatibility

• Additive at the API surface. The DocGraph work adds public names (base_blocks,
  collect, SpanRef, DocGraph, NodeModel, Node, NodeTable, NodeKind,
  Layer, Detail, Views, build_doc_graph, …) without removing or renaming any
  existing export. Section.block_type_counts() / TextDoc.block_type_counts() are
  retained; collect() is the preferred general query, not a replacement.
• Net release vs. v0.3.0. This release removes no public symbol present in v0.3.0;
  every new capability is reached through new methods, types, and exports.
  The only behavior changes an existing caller can observe are the two listed under
  Breaking Changes above: BlockType.list is now bullet-only, and default sentence
  splitting is now span-aware (boundaries may differ).

Full Changelog

v0.3.0...v0.3.1