v0.9.0 — Travelling Light

v0.9.0 — Travelling Light

The search index stops carrying a Python object per posting: postings, position maps and term counts become flat arrays over one term dictionary, and the caches become bytes instead of JSON integers. Around that, the tokenizer learns to split identifiers into the parts people actually type, pg search becomes a citizen of the shell pipeline, and caches gain rules about what may travel inside them. The loader, for its part, stops assuming a disk: every file it reads goes through a pluggable filesystem, and the host chooses which of its effects a load may use.

---

New

Pluggable loader filesystem — LoaderFS

Every file the loader reads — the entry file, lib entry points, imports, load-document, load-documents — now goes through a LoaderFS:

class LoaderFS(Protocol):
    def is_file(self, path: str) -> bool: ...
    def read_text(self, path: str) -> str: ...
    def list_files(self, root: str, pattern: str) -> list[tuple[str, int]]: ...
    def digest(self, path: str) -> str: ...

LocalFS is the local disk and stays the default, so existing callers are unchanged. Documents are read through the fs and registered as text; the engine never opens a file itself. Anything that implements the protocol — memory, a git object store, a document service — can serve a system:

system = Loader(fs=MemoryFS({"/c/main.pltg": "(fact x 1)"})).load_main("/c/main.pltg")

Host-chosen effect set — builtin_effects

Loader and LazyLoader take builtin_effects, the names of the loader's own effects to register (LOADER_EFFECTS; all by default). A host that loads sources it did not write can leave out dangerously-eval: a source that calls it fails to load, and the code never runs.

safe = [name for name in LOADER_EFFECTS if name != "dangerously-eval"]
LazyLoader(fs=fs, builtin_effects=safe).load_main(entry)

The bench speaks both

Bench, BenchServer and the technician accept fs and builtin_effects and use them for every loader they build for a bench path, for source hashing and for cache restores. A narrowed effect set is part of the source tree's hash, so a cache built under one set is never served under another.

Tokenizer v2

Identifier-like words yield the sub-tokens people type: the units between / and . (widgets_v2, inference_service) and, within each, snake/kebab/camelCase parts (widgets, v2, asset, entry), with import sigils stripped. Index and query share one splitter, and the camel split reads the original casing. A cache built by another tokenizer still loads; pg wait and pg status say so until pg reindex --force, which now re-derives every search document even when no file changed.

pg search in the pipeline

The query comes from an argument, -f FILE, -f - or a non-tty stdin. Output is path:line:text when stdout is not a terminal (or -o grep), one JSON object per line with -o json, and the grouped view on a terminal, streamed through a pager. No pagination by default; -n N with --page/--next/--prev is explicit. Opt-in highlights are served from indexed character positions.

---

Improved

Compact index

One term dictionary per index (term ↔ 32-bit id). Position maps are run-length encoded; word positions and term → line postings are CSR arrays over term ids; corpus postings map term ids to document ids; bigram/trigram tables are gone — phrase adjacency is verified against the per-line stem sequence. The caches are a JSON head plus raw array bytes in one PGZ envelope, re-based onto the live dictionary on load. Corpus-level tables are rebuilt from the per-document arrays instead of being persisted, and the heap is frozen once the corpus is loaded, so the cyclic collector no longer walks the index.

Previous-layout caches stay the operator's

A cache in the previous JSON layout is never dropped, rewritten or re-indexed on the daemon's own initiative. It is streamed into the current layout at start and served; saves are held while it is on disk. pg status names the choice, and each is an explicit command: pg cache convert, migrate, rebuild or keep.

What may enter a cache that travels

Caches embed every indexed file's text, so selection is a sharing decision. .pgignore patterns with a leading / are anchored at the root as in gitignore, and a file its own git repository ignores is never indexed — judged by the nearest repository above it. Reused entries on the pruned walk path are re-classified, so a rule added later evicts files it now covers.

Faster query paths

(in …) evaluates its document predicate once per document instead of once per posting, and (re …) narrows through the term dictionary before scanning text.

Package import

The LLM surface is re-exported lazily, so import parseltongue no longer pulls it in.

select_entries

File classification is split out of select_files, so selection runs over any listing — load-documents uses it through the loader's filesystem.

What's Changed

• roadmap: claim core-consistency-stabilisation by @sci2sci-opensource in #24
• roadmap: coverage follow-up tasks by @sci2sci-opensource in #25
• Core consistency stabilisation — clean report, zero danglings, entity imports by @sci2sci-opensource in #26
• fix(tui): query #state-tree as FocusedTree, not Tree by @PavelAbramau in #27
• fix(cli): fail with a clear message when no documents could be ingested by @PavelAbramau in #28
• Pluggable loader filesystem and effect set; compact search index by @sci2sci-opensource in #29
• Bump version to 0.9.0 by @sci2sci-opensource in #30

New Contributors

• @PavelAbramau made their first contribution in #27

Full Changelog: v0.8.0...v0.9.0