v0.6.90b420.dev729
lilbee 0.6.90b420.dev729
Extraction now runs on xberg 1.0
Document extraction moved off kreuzberg 4.x onto xberg 1.0. One extract() call now handles text, OCR, chunking, table structure and page layout in a single pass, where lilbee used to run those as separate steps with the library's own defaults.
lilbee registers its own vision model, embedder and tokenizer with xberg as plugin backends, so scanned-page OCR, chunk-boundary detection and token sizing route through the fleet you already have running.
Over 100 formats, including legacy .doc, .xls and .ppt, plus .odt, .rtf, ebooks, email and archives. Tesseract covers 100+ OCR languages, set with LILBEE_OCR_LANGUAGE="eng+deu".
Titles and authors survive ingest now. They land on the source row and on every chunk, and with embed_titles on the title is prefixed to the chunk before embedding.
At scale
Full-corpus extraction run on the migration branch, 369,758 documents and 587,676 pages of public records:
| extraction failures and panics | 0 |
| native PDF text (no OCR) | 334,605 files, 516,888 pages |
| mean | 14.2 ms/page |
| median | 12 ms/file |
| p90 / p99 | 36 ms / 213 ms per file |
New settings
| setting | default | what it does |
|---|---|---|
layout_detection |
on | Reading-order sorting and header/footer stripping for PDFs, in xberg's AUTO strategy so it only fires where it helps |
table_model |
slanet_auto |
Table structure model, used when layout detection is on |
table_extraction |
off | Indexes each recognized table as its own markdown chunk |
token_sizing |
off | Sizes chunks in real tokens from the embedder's tokenizer, not a chars-per-token estimate |
batch_extraction |
off | Coalesces concurrent extractions into one xberg batch call |
ocr_language |
eng |
Tesseract languages for the scanned-page fallback |
Intel Macs are back. xberg supports a lower macOS baseline, so lilbee-macos-x86_64 and lilbee-compat-macos-x86_64 ship again, running back to macOS 12. The compat build covers AVX-without-AVX2 machines.
Parallel processes stop stealing each other's engine ports
Two lilbee processes starting at the same moment could get overlapping engine ports, leaving one worker's client talking to another worker's engine. Each process now takes a whole 64-port block instead of an offset into a shared window.
Measured on 2x H100, one lilbee per GPU, 20k passages:
| launch | docs/s | per-card util |
|---|---|---|
| simultaneous, before | 53.9 | 8% / 87% |
| 15s stagger, before | 98.5 | 90% / 88% |
| simultaneous, after | 100.0 | 95% / 94% |
The fix beats the stagger workaround, which cost 15 seconds of dead startup per worker.
Site
Sticky mega-nav on every page. The demo and install menus deep-link into the tabs, so /#demo-crawlsite opens the clip directly. ROCm install instructions in every pane that has a ROCm build.
What's Changed
- Add ROCm install tabs and GPU backend mentions to marketing site by @tobocop2 in #640
- Give each lilbee process its own block of engine ports by @tobocop2 in #641
- Migrate document extraction to xberg 1.0 by @tobocop2 in #433
- Add a sticky mega-nav with deep links into the demo and install tabs by @tobocop2 in #643
Full Changelog: v0.6.90b420.dev728...v0.6.90b420.dev729