Skip to content

v0.2.0 — Output Quality Overhaul

Choose a tag to compare

View all tags
@KingsleyOWO KingsleyOWO released this 09 Jul 07:07
· 42 commits to main since this release
v0.2.0
aa8a3c8

Semark v0.2.0 — Output Quality Overhaul

A ground-up pass on RAG output quality, backed by a new golden-eval harness that
scores fact recall, chunk shape, and the quality gate on every run.

Reviewer (semantic-repair) — the quality lever

  • Per-role use_json_schema toggle (settings API). Turning strict
    constrained decoding off for the reviewer role lets it emit applicable
    rewrites. With strict schema on, the reviewer drops facts and its repair is
    rejected, leaving docs at needs_review; off, a local model matches a
    commercial reviewer on text-layer documents.
  • Long-output budget floored at 24000 (was 8192). Whole-document rewrites of
    multi-page docs are no longer truncated — an 8192 cap dropped ~20% of facts on
    an 8-page document. Operators can still raise it to the 32768 ceiling.
  • Text-only reviewer support (SEMARK_REVIEW_SEND_PAGE_IMAGE) for models
    that reject image input; the page-image audit is skipped accordingly.
  • Fact-preservation guard: reviewer rewrites that drop numbers, dates, legal
    references, or field labels are rejected, and the gate is re-validated after a
    repair instead of blindly clearing.
  • Reviewer client runs with no auto-retry to bound latency.

Parse & enrichment

  • Data-URI image transport by default (Ollama-compatible). The old
    static-URL default silently starved the VLM of images.
  • Per-task token caps — no more collapse-to-1024 truncation of dense form JSON.
  • ACCURATE profile uses method=AUTO — born-digital text is no longer
    needlessly re-OCR'd.
  • Scanned visual pages routed through full-page VLM enrichment.
  • MinerU hybrid backend --effort passthrough for parser A/B.
  • Higher page-render DPI, block-type coverage, and page-furniture detection.

Chunking & structure

  • heading_path metadata on structured / repair / fallback chunks.
  • CJK-aware splitting; oversized chunks split to embedder-sized (e.g. a doc that
    produced 6 chunks up to 1504 tokens now yields 19 chunks ≤ 912 tokens).

Evaluation

  • Golden-eval harness (semark-eval golden): fact-recall scoring,
    must_include / must_not_include, chunk stats, heading_path coverage, gate
    passthrough, and run-vs-run comparison.

Maintainability

  • Corpus-specific heuristics externalized to a data-driven ruleset
    (SEMARK_CORPUS_RULES_PATH).
  • Policy-title heuristic generalized — no institution hardcoding.
  • Fixtures/repo de-identified; duplicate deployment file removed.

342 tests green.

Assets 2
Loading