Pluggable file search — an engine registry, a matcher registry, and why they don't compose

[crftwr]

Prompted by #432: someone implemented Everything search in config.py — with Migemo matching that Everything itself does not have — by reaching through ctx.pane._app / ctx.pane._pane and building a virtual pane by hand. Same working method as #413, same outcome: the monkeypatch is a survey of where the extension point should have been.

Everything is not going into XeFM. I do not use Windows day to day and have never used Everything. The only test path would be a Windows machine with Everything installed, indexed, and its SDK DLL placed — and this repo has no CI at all, so every Windows check is already a hand-check with a queue in front of it. Out of tree, the DLL distribution question disappears too: voidtools ships the SDK separately from the app, so nothing would be bundled and nothing would need signing (Windows has no equivalent of macOS library validation; a full-trust MSIX loads an unsigned DLL from an absolute path fine).

What goes in is the extension point. Everything becomes a worked example on the wiki — not because there is nowhere else (#426 §7.3's question has since been answered in the tree, by doc/VIRTUAL_FOLDERS_FEATURE.md) but because this one should not be in the tree at all: a page outside it does not imply support, and can be edited by the person who actually runs it.

There is a second, cheaper answer to #432 that is not a plugin API at all — letting a list of paths from any source become a pane. That is split out into #453; everything below is about engines driven from XeFM's own dialog.

The interesting part is that this splits into two registries that look composable and are not.

---

1. This is a different door from #426

#413 wants a namespace you can navigate into. #432 wants a flat set of pointers to files that already exist. They meet nowhere.

	#413 → #426	#432
Wants	reg://HKEY_CURRENT_USER/…, a hierarchy	a flat result set of real paths
Internal door	Path._create_implementation (path.py:1086-1121)	pane["virtual"] (app.py:4337-4340)
Third party writes	MockPathImpl = 384 lines, 61 methods (#426 §2)	today: a hand-built 5-key dict
Needs a new PathImpl	yes	no — the results are ordinary local Paths

#426's tracks have since landed — xefm/path_base.py's ReadOnlyPathImpl, PATH_SCHEMES, the capability set — and none of it makes #432 one line easier. Nothing below depends on #426, or the reverse.

---

2. Two registries, and why they do not compose

§5 of #378 already lists Search matchers — name → (pattern, text) -> spans, with Migemo special-cased at search_match.py:48. That row is about judging a candidate. What #432 needs is producing candidates. Two rows, not one:

row	contract	who walks
Search matchers (already in the table)	(pattern, text) -> spans	core walks, the plugin judges
Search engines (new)	(root, query, options, cancel) -> Iterator[EntryInfo]	the plugin walks, core displays

My first instinct was that the engine row would consume the matcher row — hand Everything the Migemo regex. That is true for Migemo and false in general, and the distinction matters:

A matcher is a Python callable. An external engine accepts only a string in its own language. Migemo rides Everything because migemo_search.get_regex() happens to expose a compiled regex string. That is a property of the implementation, not of being a matcher. fnmatch.fnmatchcase(name, pat) (search_options.py:65) is a function call with no serialized form. A user's fuzzy or phonetic matcher has none either.

So matcher × engine is not a free cross product but a table full of holes. Measured against Everything:

Migemo's generated regex
kensaku	75 chars — (?:(賢作|謙作|…|検索|…|健[策作]|けんさく|ｋｅｎｓａｋｕ|kensaku)|ケンサク|ｹﾝｻｸ)
nihongo	37 chars
TenkiYohou	172 chars — two alternation groups concatenated

Only |, (...), (?:...), [...] and literals. No quantifiers, no anchors, no backreferences. Everything uses PCRE, so all of it passes — though (?: is not in the documented operator table, and | is eaten by the term parser unless the pattern is quoted: regex:"(検索|けんさく|kensaku)". Note also that the concatenation in TenkiYohou demands adjacency, which Everything's native |/AND syntax cannot express — so regex is the right tool for multi-word queries and native OR is faster for single words.

fnmatch, by contrast:

fnmatch	Everything's normal syntax
* ?	supported (*, **, ?)
whole-name rule	identical — "The whole filename is matched when using wildcards" is exactly #231's exact-glob rule
[abc] [!abc]	not supported. Everything documents three wildcards and no character classes

The character that breaks translatability is already singled out in the tree: migemo_search.py:62, _GLOB_CHARS = ('*', '?', '[').

The honest model is that the two registries are alternatives at the same level, not two axes of one product. Every consumer of matchers today — search_match.compile_query/hit for the pane filter and the filter dialogs (file_list_manager.py:724,737, filter_list_dialog.py:166-188), filename_matcher for the walk, find_spans for the viewers — is an in-process, candidate-at-a-time consumer. An engine replaces in-process matching entirely.

The one bridge worth building is optional by construction: a matcher may declare to_regex(pattern) -> str | None. Undeclared means absent. Only the matchers an external engine can host volunteer for it, and nothing breaks for the ones that cannot — the capability-set shape #426 §5 argued for and path_base.py:448 now ships (CAPABILITIES = frozenset({...})).

---

3. The engine registry

The threading contract is already made

#378 §5 wanted the worker-thread promise settled with the first registry. It has been: user_api.py:56 states that actions and hooks run on the UI thread, synchronously, and :270 that a PATH_SCHEMES listing runs on a worker thread, exactly as ordinary navigation does. So #378's open question 4 is answered, and answered the way this row needs.

What this row adds for free is the generator form of it. progressive_search_dialog.py:13-17 already runs search_iter(mode, query, cancel) on a worker, flushes in batches, and supersedes a generation on the next keystroke. Nothing has to be promised that is not already running.

It is also the heaviest contract in the table. Everything else there is a pure function called once. This one is a generator that must yield incrementally, check cancel between yields, tolerate being superseded, and never block. A bad provider hangs the search dialog (not the UI thread, but the results never arrive). It is the only row where an implementation that ignores the contract produces an observable hang.

The dialog is already mode-generic, except in two places

Everything keyed on mode is already opaque: options.visible(mode) → Option.applies(mode) (options.py:126-128), _BATCH.get(mode, 50) (progressive_search_dialog.py:197), titles.get(self.mode, "Search"). The only hardcoded binary is _switch_mode (progressive_search_dialog.py:303-305) and the label naming the mode Tab switches to (:525). "Flip a pair" becomes "cycle a list".

pane["virtual"]["kind"] was written for this and has never been read

_feed_search_results sets "kind": "search" (app.py:4338). Nothing anywhere reads it — the same shape as the ten capability methods with no callers in #426 §5. A second kind is exactly what #432 is. Without it the header says ⌕ "query" — N results (filename), because app.py:789-791 reads virtual["mode"] directly.

What this removes

#432 needs no pane-feed API. If the engine is a registry row, the provider returns paths and core's existing on_accept → _feed_search_results does the feeding. Nothing about pane["virtual"] is published; ctx.pane._app stops being necessary. I had thought a PaneApi.feed() would be the short path — it is not needed at all, which is a better outcome than publishing a dict shape that has already grown meta once (for content search) and root once (#383).

The contract yields entries, not paths

Bare Paths would make the pane stat every result — for a machine-wide set that is the whole cost of the feature. filters.py already promises the opposite for a pane's own listing: "size and mtime are free, seeded from the listing's own attribute record". An engine usually has them already (Everything3_AddSearchPropertyRequest returns size and dates), so the contract should yield EntryInfo — or (path, attrs) — and fall back to statting lazily only where a provider has nothing. Same shape listdir_attrs was added for. It also decides whether a config-defined FILTERS predicate over an engine's results is free or ruinous.

What does not go away: root

Everything is machine-wide; the dialog's root is the active pane's directory. root is load-bearing — relative-name display (file_pane.py:358), sort rel_root, the cwd for external programs (app.py:4627), app.py:6607. A result set spanning drives has no common ancestor. A provider has to be able to say so, and core has to decide what that means (full paths, and the cwd falls back to where?). Small, but it is the one genuinely new decision.

---

4. The matcher registry

This one is easier than its place in the table suggests, because the contract has already been chosen twice.

PuiKit already has the slot. json_view.py:273, markdown_view.py:1065, table_view.py:178 all carry

self.search_matcher: Callable[[str, str], list[tuple[int, int]] | None] | None = None

and text_viewer.py:1190 assigns migemo_search.find_spans straight into it.

The additive rule is already law on both sides. markdown_view.py:1780 names the return value extra — something added to its own literal pass. migemo_search.py's header says it outright: "an engine quirk can only ever add matches, never remove them." Making that the registry's rule dissolves most of the hard questions: composing N matchers is a union, order and precedence do not arise, and a broken matcher can only add noise. None already means "does not apply", and every failure path in migemo_search already returns it.

What is actually hard, in order:

(a) under_gate, and monotonicity. dead_end (#370, search_match.py:72) assumes typing only narrows. Migemo violates that — ni finds nothing, nih finds 日本 — so under_gate carves out the exception. Any custom matcher can violate it in its own way and core cannot know how. The contract needs may_widen(pattern) -> bool, which is an odd thing to ask of a third party, and getting it wrong means isearch refuses keystrokes it should accept — a symptom nobody will attribute correctly. Default it to True: dead-end detection switches off while a matcher that has not answered is registered. A nicety lost, never a wrong answer. #426 §4's rule, that the published base is where leniency belongs, applies verbatim.

(b) The contract is two-phase, and #378's table writes it as one. search_match.py's header is explicit that compiling is the expensive step (seconds for 1–2 characters) and matching is cheap, and that callers hoist compile_query out of the row loop. find_spans survives the one-phase signature only because get_regex is @lru_cache(maxsize=256). Published as (pattern, text) -> spans, every implementer independently rediscovers that they need a cache. It should be compile(pattern) -> compiled | None plus spans(compiled, text), with core adapting to PuiKit's one-phase slot.

(c) Which string, and where do the spans index. search_nfc/match normalize the haystack; find_spans deliberately does not, because normalizing shifts the offsets the spans index into. Get it wrong and highlights are off by one, on macOS, only for decomposed names. Impossible to guess, trivial to state once.

(d) The non-ASCII filter is Migemo-specific and must stay out of the contract. find_spans keeps only spans containing a non-ASCII character, because Migemo's own expansion includes the raw romaji and would double-claim what the caller's literal pass already found. A fuzzy or phonetic matcher has no such problem. The temptation is to copy the signature along with the filter.

The de-hardcoding is small. Migemo is named in exactly two places: search_match.py:53 bakes migemo_search.get_regex(token) into Token, and :67 calls migemo_search.search_nfc. Token becomes (glob, [compiled per matcher]) and hit becomes native or any(...). Ten lines in one module, plus pointing the viewer call sites at the same lookup.

---

5. Options: each participant declares its own

The split this needs is already written in the tree, twice. search_options.py's header:

Four options, and they are two different kinds of thing: how the query is read — case, word, regex; what is walked — subdirs.

and Option.persist (options.py:47):

Options that change how a query is read persist for the session; options that change what is walked do not.

That is the matcher/engine split under different names. It has been sitting there waiting for owners, because with one engine there was nothing to own.

Invert the declaration

Today the option names its modes — Option(REGEX, …, modes=("content",)) # filename search speaks glob, not regex. That comment is already an admission that different modes speak different query languages. Tomorrow the participant declares its own options, next to the code that interprets them — which is what search_options.py already does for core (it holds content_regex and filename_matcher in the same module as the declarations):

class EverythingEngine:
    OPTIONS = (
        "case",                                     # ride the existing definition
        "word",
        Option("index", "Whole machine", flag="Idx", default=True),
    )

A bare string means "I interpret the one that already exists" (warn and ignore if nobody defined it, as EVENT_HOOKS already does). An Option means "here is my definition; if the name is taken, only the interpretation counts." One field, two forms, no central table to drift from.

Visible set = the union of what the participants in this search declare. modes=("content",) was that relationship hardcoded on the wrong side.

Identity is the name; first declaration wins

accel_map (options.py:173-182) already establishes exactly this rule for the accelerator letter — "First declaration wins, so a duplicated initial degrades to 'the second one has no letter'". Widening it from the letter to the whole declaration is consistent.

It is deliberately the opposite of the last-registration-wins rule #378 §5 cites for ACTIONS, and it should stay that way: an action is a replacement — overriding core is the point — while an option definition is shared vocabulary, and a plugin silently relabelling core's "Case sensitive" is a bug. Worth stating before the one mechanism flattens both.

Three things fall out

There is never more than one interpreter per option. Enumeration options belong to the engine, because only the engine walks. Query-reading options belong to whoever does the matching — core's matchers when it is core's walk, the engine itself when it is external. Because engines and matchers do not compose (§2), those are never both true. The union-or-intersection question never arises.

persist becomes derivable. The owner is the category: an engine's options are enumeration options, a matcher's are query-reading. Today the correlation is maintained by hand.

subdirs disappears correctly under Everything, with no new rule. Not "Everything declined to name it" but "nobody in this search defines it" — and Option's existing principle already says what that looks like: "A gated option is not dimmed but absent: an option that cannot do anything must not look like one that is merely off."

One rule that has to be added

Matchers are additive, but options narrow. If matcher B does not interpret case, turning case-sensitivity on does not remove B's hits — the option silently fails.

A matcher sits out any search in which an option it does not declare is set to a non-default value.

"Non-default" is the load-bearing part: a matcher that declares nothing still works in the default configuration and drops out only when the user has explicitly asked for something it cannot do. It removes additive contributions, never adds wrong ones, and needs no judgement about whether an option narrows or widens (regex on/off does both).

Two landmines an open set steps on first

The accelerator collision finally fires. The mechanism exists and degrades safely, but silently — the second option just has no letter. Four hand-tuned labels ("the four labels have to start apart") never trip it; plugins will, and the author has no way to find out. accel_map should warn. One line, and it belongs with the first registry.

The chip strip has no cap. progressive_search_dialog.py:452-466 pins the strip to the right of the status row and elides the status text to make room (avail = wu - 4.0 - (strip_w + gap)). Ten chips eat the status line; on a narrow terminal they eat all of it. Four fit by construction today.

---

6. Conditions that are not booleans

Everything's query language carries value-bearing terms — size:>1mb, dm:today, ext:mp4;mkv, with !, | and <> around them. None of that fits §5, where every option is on or off by construction. It does not have to, and the tree already says why.

filters.py faced the same question for the pane filter and answered it:

"show me what changed today", "show me anything over 100 MB", "show me the images" are all questions about an entry rather than about its name, and a glob cannot ask them. Rather than grow a pattern language for each, this exposes the predicate.

And options.py excludes the case explicitly: "Where a setting has three answers, the surface owns three declared options or a picker of its own — it does not go here."

So there are three tiers, not one:

tier	whose	example	machinery in XeFM
Query terms	the engine's query language	size:>1mb, dm:today	none needed — the query field carries them
Options	binary modifiers	case, word	§5
Predicates	Python callables over EntryInfo	a config's "big" filter	FILTERS, already shipped

Everything's size and date conditions are tier 1. They are typed into the query field, in the engine's own language, and §5's rule that an engine owns its query language end to end already covers them. Nothing new is required — but it has to be said, or someone will try to model size: as ten boolean chips.

The non-composition of §2 is broader than matchers

"Engine × matcher" was too narrow. The real statement is engine × anything expressed as a Python callable — and two of those already ship:

	with an external engine
Sort keys	fine. Applied in-process to the fed result set, which flm.refresh_files already does
Filter predicates (FILTERS)	fine. The pane filter applies after feeding, in memory
Matchers	no. Matching is the search; applying it afterwards means the engine has already decided

So size and date conditions are also reachable the other way — import the results, then apply a FILTERS predicate. Except that _RESULT_CAP = 1000 makes that quietly wrong: "over 100 MB" applied to the first thousand results of a machine-wide index is not "over 100 MB". Either the cap becomes something an engine declares, or narrowing has to happen inside the engine — which is another argument for pushing value-bearing conditions into tier 1 and leaving them there.

What this asks for, concretely

Not a criteria registry. Two small things:

1. Say in the contract that the engine owns its query language including value-bearing terms.
2. Give the engine a way to say what that language is — a hint line or placeholder in the dialog, next to the per-mode titles that already exist. That is the actual gap, and it is cheap.

If a plugin ever needed a picker of its own, note that a plugin-contributed picker is plugin-contributed UI — the second breach of the façade after VIEWER_RENDERERS, which #378 §5 counts as the sole exception. The way out is the one Option already takes: declare it as data and let core render it.

7. Corrections this implies for #378 §5

1. The Search matchers row's contract shape is wrong: (pattern, text) -> spans is the one-phase form, and compiling is the expensive half. It also needs a note that the row serves in-process matching only and does not ride an external engine.
2. Add a Search engines row.
3. Add a third cross-cutting decision, beside "one mechanism, not ten register_*" and the threading contract:

Registries are not composable by default. Where two of them can interact, the interaction has to be designed, not assumed.

Most rows never meet — an image decoder and a sort key are independent forever. Engines × matchers is the first pair in the table that looks composable and is not, and that is the kind of assumption that gets made silently and breaks later.

---

8. Order

		depends on
①	Per-participant option declaration + accel_map warning + chip cap	—
②	Matcher registry (de-hardcode search_match.py:53,67, may_widen defaulted true, two-phase)	①
③	Engine registry (_switch_mode → cycle a list, kind in the header, the root decision)	①②
④	Everything provider on the wiki, out of tree	③

① first because both registries need to declare options and neither should ship a second option mechanism. ② before ③ because the matcher contract is the one that is already settled — it is the cheaper shake-down of the machinery, and §2 means ③ does not depend on it semantically, only on the option plumbing.

Against #378's open question 5 — which registry goes first — matchers have a property neither image decoders nor sort keys have: the contract, the failure mode, the additive rule and the meaning of None are all already decided and running in two codebases. The design risk is close to zero. Sort keys still win on contract size.

---

Open questions

1. root for a result set with no common ancestor. Full paths in the name column is obvious; what the external-program cwd (app.py:4627) falls back to is not.
2. Option name namespacing. "case" being shared vocabulary makes the name space effectively global. Reserve core's four and recommend a prefix (everything.index) for plugin-specific ones — or start with "collide and warn", as accel_map already does, and tighten later?
3. How an engine is selected. Options are on/off by construction ("Where a setting has three answers, the surface owns three declared options or a picker of its own"), so the engine cannot be one. Tab cycling a list of three is fine; a list of eight is not.
4. Does a matcher ever get to see the candidate's EntryInfo, or only its name? Everything in §4 assumes a string. A matcher that wants to match on size or date is really a filter predicate — a different row in Widening the image viewer's format support — in-memory raster data first #378's table — but the line will get tested.
5. REGEX may be a matcher selector wearing an option's clothes. modes=("content",) plus "filename search speaks glob, not regex" describes a choice between two matchers, not a modifier. Probably not worth changing shipped behaviour over, but it is worth knowing before the vocabulary is published.

[crftwr]

Update: #453's path shipped, and it reverses one conclusion here

#453 stages ①–③ are in (#492–#499). A pane can open a list of paths from the clipboard, from a command's output, or from a list file (Go → Open List). The command source covers the Everything case of #432 with no plugin API: es.exe -path . ext:py.

§3's "no pane-feed API" no longer holds. That conclusion depended on the engine registry existing to do the feeding. With the registry on hold, the feed is the extension point, so it is published: PaneApi.open_list(paths, *, title). It is kept narrow, as argued there. It takes paths (strings, Paths or entries, absolute or relative to pane.path, URIs included) and a title. The pane["virtual"] dict is not published. A config's search runs on the UI thread like any action; which paths exist is checked on the listing worker.

Other items from §3 are settled in the same work:

• virtual["kind"] is read now. A list reads [title] root in the header, and a search keeps its ⌕ banner.
• root with no common ancestor (open question 1). The common ancestor is taken from the surviving paths. With none, the sentinel name_key.WHOLE_PATH names each row by its whole path, so None keeps meaning basename. The external-program cwd falls back to the directory the pane showed before.
• Attributes, not bare paths. This is not done: open_list takes paths, and each is stat'd once on the worker. Accepting EntryInfo can be added later without breaking anything.

The registries stay on hold. What would bring them back is a need the list sources cannot meet. The two in view are results streaming into XeFM's own dialog as you type, and Migemo over an external index. The matcher registry (§4) is untouched; its contract is as settled as it was.