🧱 Rebar — rebuild the hierarchy of Japanese documents from flat text

Restore the heading hierarchy that text extraction throws away — from plain text alone — and prove that structure-aware chunking actually improves retrieval. Runs on CPU. A few kilobytes. Layout-independent. A different tool from large multimodal PDF parsers.

When a PDF (or any document) is turned into text, the heading tree — 第1章 → 1. → (1) → ア — collapses into an undifferentiated stream of lines. Rebar reconstructs that tree from the plain text and then chunks along it, so an article stays with its proviso, a table with its title, and a FAQ question with its answer. For RAG, that means retrieval returns complete answers instead of fragments.

• No PDF layout analysis. Rebar works on the text after extraction. Layout hints can be used as an auxiliary signal, but everything works without them.
• Not a large multimodal model. The whole pitch is the opposite: local, cheap, layout-independent.

---

The idea: depth is relative

The same marker means different depths in different documents. 1. is top-level in a memo but second-level under 第3章. So depth can't be decided per line — Rebar looks at the whole document and infers the numbering scheme jointly (document-level structured prediction).

flowchart LR
  subgraph A["memo (no chapters)"]
    a1["1. Purpose"] --> a2["depth 1"]
  end
  subgraph B["regulation (under 第3章)"]
    b1["1. Purpose"] --> b2["depth 2"]
  end
  classDef n fill:#dcebe1,stroke:#2f6f4f;
  class a2,b2 n;

Loading

Depth is not hard-wired to a marker kind; it is derived from the nesting evidence in that document. Rebar uses the Japanese public-document standard (第1→1→(1)→ア→(ア)) and the statutory chain as a prior, but when a document's nesting contradicts the prior, the evidence wins — which is exactly what lets Rebar generalize to numbering schemes it never saw in training.

Pipeline

flowchart LR
  T["flat text<br/>(damaged OK)"] --> M["MarkerInventory<br/>marker kinds (depth-free)"]
  M --> C["HeadingClassifier<br/>which lines are headings"]
  C --> S["DocumentSchemeInference<br/>kind→depth for THIS doc"]
  S --> B["TreeBuilder<br/>constrained decode + repair"]
  B --> K["StructureAwareChunker<br/>chunk along the tree"]
  K --> R[("RAG / retrieval")]

Loading

---

Headline results

All numbers are on the reproducible synthetic corpus (python benchmarks/run_benchmarks.py).

1. Known failure modes drop to zero

Fixed-length (A), recursive-character (B), and semantic (C) chunking regularly split an article from its proviso, a table from its title, or a FAQ Q from its A. Rebar (D) cuts on tree boundaries, so it splits none of them.

Chunking strategy	Article↔Proviso split	Table↔Title split	FAQ Q↔A split
A · fixed-length	20.5%	2.7%	16.6%
B · recursive-character	24.1%	1.0%	13.7%
C · semantic	95.2%	97.4%	97.0%
D · Rebar	0.0%	0.0%	0.0%

2. Depth accuracy generalizes to unseen numbering schemes

Set	Heading F1	Depth accuracy	Tree agreement (parent)
Test (seen schemes, clean)	0.997	1.000	1.000
Hard (UNSEEN schemes, clean)	0.995	1.000	1.000
Test (damaged)	0.53	0.81	0.59
Hard (unseen, damaged)	0.54	0.35	0.52

Damage degrades results honestly. The hardest quadrant — unseen scheme × heavy damage — is where future work lives; damage-augmented training (--augment-damage) improves it.

3. Downstream: the whole answer lands at the top

Complete-answer recall@1: the top-1 retrieved chunk contains the entire unit (article + proviso, or table + rows), not just a fragment.

Strategy	Article+Proviso	Table+Rows	FAQ Q+A	Fragment recall@3
A · fixed-length	63.1%	64.3%	55.2%	0.944
B · recursive	60.9%	65.3%	56.8%	0.928
C · semantic	0%	0%	0%	0.170
D · Rebar	71.7%	96.6%	85.2%	0.958

---

Quickstart

pip install -e .          # core depends only on numpy
pip install -e ".[full]"  # + training / export / Space / figures

CLI:

rebar outline doc.txt              # print the restored hierarchy
rebar scheme  doc.txt              # show the inferred kind→depth + nesting evidence
rebar chunk   doc.txt --max 800    # print structure-aware chunks

Python:

from rebar import Rebar
text = open("doc.txt", encoding="utf-8").read()
print(Rebar().outline(text).outline())     # restored tree
chunks = Rebar().chunk(text, max_len=800)  # structure-aware chunks

Drop-in for LangChain (3 lines)

from rebar import Rebar
from langchain_core.documents import Document
docs = [Document(page_content=c.text, metadata={"path": c.path, "marker": c.marker})
        for c in Rebar().chunk(text, max_len=800)]

Drop-in for LlamaIndex (3 lines)

from rebar import Rebar
from llama_index.core import Document
docs = [Document(text=c.text, metadata={"path": c.path}) for c in Rebar().chunk(text, 800)]

---

The four cores

Module	Role	Claim it proves
rebar/markers.py — MarkerInventory	Systematic vocabulary of Japanese numbering; extracts a marker's kind + ordinal only (never depth)	generalization / speed
rebar/scheme.py — DocumentSchemeInference	Infers kind→depth for the whole document; public-document standard as prior, overridden by nesting evidence	generalization / depth accuracy
rebar/tree.py — TreeBuilder	Constrained decoding (depth +1 at a time, siblings share a scheme, numbers roughly consecutive) with repair	depth accuracy
rebar/chunker.py — StructureAwareChunker	Chunk along the tree: include headings, never split proviso/table/FAQ, rules for over-length nodes	downstream retrieval

Every function's docstring states which claim it substantiates (depth accuracy / generalization to unseen schemes / downstream retrieval / speed).

---

Data construction (the core of the project)

1. Collect documents with explicit structure (Markdown / HTML / statutory XML / public regulations). Check the license and redistribution terms; they are recorded in the dataset card (scripts/build_dataset.py emits a provenance table).
2. Flatten — drop markup but keep the visible numbering (rebar/flatten.py). Record each heading's line position and depth as the gold label.
3. Damage layer (rebar/damage.py) — reproduce text-extraction reality: mid-line breaks, multi-column line-order mixing, header/footer contamination, inter-character spaces, ruby injection, broken tables, page numbers, full/half-width jitter. Deterministic given a seed.
4. Split — by document (no document straddles train/test) and separate a hard set of held-out numbering schemes (rebar/dataset.py).

Redistribution care: --redact ships only the structure labels plus minimal text fragments, never full source bodies. The bundled synthetic corpus is CC0, so the whole repository reproduces every result on its own.

Reproduce end-to-end:

python scripts/build_dataset.py --n-per-scheme 80              # build HF dataset to disk
python scripts/build_dataset.py --push NagaYu/rebar-structure  # push to the Hub (needs login)
python scripts/train.py  --n-per-scheme 60                     # train the classifier
python scripts/export.py                                        # export ONNX / GGUF
python benchmarks/run_benchmarks.py && python figures/make_figures.py

---

Deliverables

• models/rebar_heading_clf.onnx (~0.6 KB) — the heading detector as sigmoid(x·w+b), verified against numpy at export time.
• models/rebar_heading_clf.gguf (~0.7 KB) — a spec-compliant GGUF weight container (architecture rebar-logreg). This is not a llama.cpp language model; it is an honest container for a logistic regressor's weights + feature metadata.
• app.py — a Gradio Space: paste text, see the restored tree, and view Rebar's chunks next to what fixed-length chunking would have produced.
• scripts/ (build_dataset / train / export), benchmarks/, figures/.

Hugging Face: Space · Model · Dataset

---

Evaluation methodology

• Structure: (1) heading-detection F1, (2) hierarchy depth accuracy (always reported on the hard/unseen set), (3) tree agreement (parent-relationship accuracy), (4) robustness clean vs. damaged. Predicted↔gold headings are matched by content (marker + title), not line index, because damage reorders lines.
• Downstream (the main claim): an auto-generated QA set including structure-crossing questions ("what is the exception in Article 3?"), compared across A/B/C/D. Metrics: retrieval recall and rank, the failure-mode split rates above, and complete-answer recall@1 (does the top chunk contain the whole answer).

---

Limitations

• No PDF layout analysis — Rebar is the stage after text extraction.
• Unnumbered headings (e.g. Markdown headings without numbering) are hard, because depth is inferred from marker kind; Rebar targets numbered Japanese documents.
• Heavy damage that wraps heading lines themselves lowers detection recall — reported honestly rather than hidden.
• Always verify the redistribution terms of any real corpus you add; the bundled synthetic data is CC0.

License

Code: Apache-2.0. Synthetic data: CC0. Ingested real documents follow their own source licenses.

Citation

@software{rebar_ja,
  title  = {Rebar: Hierarchy restoration for flattened Japanese documents},
  author = {NagaYu},
  year   = {2026},
  url    = {https://github.com/NagaYu/rebar}
}