This repository was archived by the owner on Oct 22, 2026. It is now read-only.
Repository navigation
v0.2.0
The format discovery engine, a direct Arrow path for Parquet, and a
per-model format selector in ldf-mcp. The specification stays v0.1;
two encoder bugs are fixed against it.
Added
ldf-mcp --model ID(inproxy,clickhouseandfiles): each result
goes out as LDF only when measured evidence for that exact model on data
of the same shape shows no accuracy loss against JSON (at least 50
questions, no more than 3 points below, no significant loss, no more
rows than measured) and LDF is at least 5 percent smaller; otherwise
JSON, with the reason on stderr. A model without evidence gets JSON
throughout and no LDF primer.--evidence FILEreplaces the built-in
evidence. Without--modelnothing changes.ldf-select, a new crate with the selector's rule and the evidence
table built in, shared byldf-mcpandldf-discover.ldf::arrow(featurearrow): encodes Arrow RecordBatches directly,
byte-identical to the record path, 1.9x to 3.2x faster in block form at
1M rows and 1.4x to 4.3x less memory.ldf-mcp convertuses it for
Parquet and falls back to the record path for lists of structs.
ldf_data::input::ColumnReaderis public for it.ldf-discover(not published): a parameterized encoder whose layout
choices are knobs, a decoder for every point, Stage A token search over
the space, and theselectcommand.- Evaluation: four new task kinds, more questions per kind, held-out
fixtures, a bare answer outcome, Stage B and the DeepSeek sparse sweep
(reports underdocs/reports/),eval/analysis/evidence.pyfor the
selector's evidence, andeval/bench_mcp.pyfor end-to-end MCP timing.
Fixed
- An object field that is null in some rows decoded as absent; such a
field is now ajcolumn (spec 7.1). - A bare string starting with
@as the first cell of a row was read back
as a header; it is now quoted (spec 5). LDF output on the twelve medium
fixtures is byte-identical before and after both fixes.