User-defined virtual folders — a scheme registry, and the base class that has to ship with it

[crftwr]

Prompted by #413: someone wants to browse the Windows registry in a pane — read-only, with the actual editing handed off to regedit — and asked whether config.py can define a virtual folder the way cfiler does. They then had an LLM produce a monkeypatch, which works, and which found every place I would have had to fix anyway.

Scheme registration is the visible ask, and it is the cheap half. The half that decides whether this is usable is what a third party has to write once they have registered, and what I am promising not to break afterwards. Same shape as #396 and #378: the mechanism underneath is the thing worth getting right.

1. How it works today

Path (xefm/path.py) is a façade that delegates to a PathImpl, chosen in _create_implementation (path.py:1086-1121) by an if-chain on the URI prefix. The contract there is already uniform and already right: a URI string goes in, a PathImpl comes out — ArchivePathImpl(path_str), S3PathImpl(path_str), SSHPathImpl(path_str). Turning that into a table lookup is not a redesign.

The scheme list itself is hardcoded in three places, and #413's monkeypatch found all three:

• path.py:1077 — the guard that keeps a URI away from PathlibPath
• app.py:3104 — XeFMApp._REMOTE_SCHEMES, which Jump to Path resolves against
• config.py:1178 — _REMOTE_SCHEMES, for DRIVE_LOCATIONS

Patch only the first and Jump to Path treats reg://… as a path relative to the current directory and reports "Path does not exist". That report is accurate.

Two smaller things fall out of looking. scp:// and ftp:// sit in all three lists with no branch in _create_implementation, so they quietly resolve to LocalPathImpl. And the config side needs no new machinery: xefm/user_api.py already loads, validates and reloads ACTIONS / EVENT_HOOKS / SORT_KEYS / FILTERS declaratively, warning per entry and never failing the whole config. PATH_SCHEMES is one more table in that same pass.

2. Registration alone gives #413 nothing

PathImpl has 61 methods. 60 of them are abstract; listdir_attrs (path.py:267) is the only concrete one, and it is concrete only because it is written in terms of iterdir.

The measurement already exists in the repo. test/test_mock_storage_extensibility.py exists to prove that a new storage type needs no UI changes, and to prove it, it writes MockPathImpl — 384 lines, 61 methods, for a backend that keeps three files in a dict. That is what I would be handing someone who wants to list registry keys.

3. What can be defaulted, and on what grounds

Two different justifications get mixed together here, and separating them decides the class layout.

parent and joinpath across the three remote impls reduce to the same operation:

	root prefix	key
S3	s3://bucket/	key/path
SSH	ssh://host	/abs/path
Archive	archive://<file>#	internal/path

Split on /, drop or append a segment, re-join with the prefix. The differences are the prefix spelling, the trailing-slash convention, and behaviour at the root — where archive is genuinely different, because the parent of an archive's root leaves the archive. Two hooks (_root_prefix(), _with_key()) cover the first two, and the third is exactly what an override is for.

By group, then:

group	n	grounds for a default
String arithmetic — name parent parts joinpath with_* relative_to glob __eq__ …	27	The implementation is genuinely the same
Write operations — mkdir unlink rename write_bytes chmod …	11	Not shared code — shared policy (read-only ⇒ raise)
Capability / display — supports_* get_search_strategy get_display_* …	12	A safe answer exists; also policy, not code
Core I/O — exists is_dir iterdir stat open …	10	No default possible

Which argues for two layers rather than one class with everything filled in:

PathImpl (ABC)
  └─ UriPathImpl            27 shared implementations; the rest still abstract
       └─ ReadOnlyPathImpl  write ops raise; capabilities take conservative answers
            └─ a user's class    8–10 methods

Keeping them apart matters because UriPathImpl is not read-only-specific. s3.py and ssh.py carry roughly 290 lines between them implementing those same 17 arithmetic methods twice — a debt UriPathImpl can eventually absorb, and could not if the same class had also decided that writing raises.

De-abstracting PathImpl itself is the one-class version of this, and it costs the check that currently catches an internal impl forgetting a method at import time. The ABC should stay strict; the published base is where leniency belongs.

4. Is PathImpl stable enough to publish?

Signatures, yes — not one has been changed or removed, ever. Growth, no.

	methods	change
2025-09-27	—	PathImpl added (then src/tfm_path.py)
v0.98 (2025-11-25)	52
v1.0.1 (2026-07-24)	60	+8 display / strategy / capability methods
v1.0.4 (2026-08-02)	61	+listdir_attrs
v1.4.0 → today	61	—

48 of the 61 are names pathlib.Path also has, which is why they do not move — that half is pinned to the stdlib rather than to my judgement. 13 are XeFM's own, and every one of the nine additions above landed in those 13. listdir_attrs arrived because the async listing work wanted a per-backend hook. Nothing about that pattern has finished.

The failure mode is not a signature change. It is a new @abstractmethod: the moment one lands, every external subclass raises TypeError at import, and the user's entire config stops loading. Six quiet weeks is not a track record.

So the published extension point cannot be PathImpl. If it is ReadOnlyPathImpl, a future method arrives carrying a default and outside code never notices. The intermediate class is the compatibility buffer; the ergonomics are a side effect. Worth a test that fails when a new abstract method has no default in the published base, so the rule outlives my remembering it.

5. The capability methods, and the ten that nothing calls

These are compile-time constants:

method	Local	S3	SSH	Archive
supports_write_operations	True	True	True	False
supports_directory_rename	True	False	True	False
supports_file_editing	True	False	False	False
supports_streaming_read	True	False	False	False
requires_extraction_for_reading	False	True	True	True
should_cache_for_search	False	True	True	True
get_search_strategy	'streaming'	'buffered'	'buffered'	'extracted'
get_display_prefix	''	'S3: '	'SSH: '	'ARCHIVE: '
is_remote	False	True	True	delegates

Only ArchivePathImpl.is_remote (archive.py:2312) varies per instance, by asking the archive file's own path.

And then the part I did not expect. supports_directory_rename, supports_file_editing, supports_write_operations, requires_extraction_for_reading, supports_streaming_read, should_cache_for_search, get_search_strategy, get_display_prefix, get_display_title and get_extended_metadata have no callers in the application at all. One abstract declaration, four implementations, one façade delegation at path.py:1379-1418, some tests — and nothing anywhere that reads the answer. The file list stats directly (file_list_manager.py:537). doc/dev/PATH_POLYMORPHISM_SYSTEM.md still shows TextViewer / InfoDialog / SearchDialog consuming them; that diagram is stale. What is live is is_remote, get_scheme and listdir_attrs.

Publishing the interface as it stands would mean asking an outside implementer ten questions and discarding every answer.

So they collapse into a declaration instead of ten methods:

class RegistryPathImpl(ReadOnlyPathImpl):
    SCHEME = "reg"
    CAPABILITIES = {"streaming_read"}    # undeclared == absent
    SEARCH_STRATEGY = "buffered"
    def is_remote(self): ...             # per-instance; default from a class attribute

Six flags become a set, three become plain values, and three stay methods because they compute something per instance (is_remote, get_display_title, get_extended_metadata). The set form is forward-compatible by construction — a capability added later is simply undeclared by classes written today — which takes this group out of §4's hazard entirely. Unknown names warn and are ignored, exactly as EVENT_HOOKS already does.

6. Four tracks, and the order

	track	touches	notes
②	UriPathImpl / ReadOnlyPathImpl	one new module	Independent of everything. Acceptance measure: MockPathImpl's 384 lines become ~10 methods
③	Capability consolidation	PathImpl + the four impls	Independent. Low risk precisely because ten of them are dead
①a	Scheme registry	path.py, app.py, config.py	Internal cleanup, no promises made. Also settles scp:// / ftp://
①b	PATH_SCHEMES config surface	user_api.py	The Alpha promise. Needs ①a

No two of them edit the same region, so they are separate PRs that will not conflict. Alpha needs all four plus a sample and docs, but each lands on its own merit before then. user_api.API_VERSION is already 0 for this exact reason, and the same Preview framing covers this.

② first is deliberate: publishing PATH_SCHEMES before there is a class worth inheriting means advertising a registration point for something nobody can write. ③ before ①b for the same reason — changing what a capability means after publication is the breaking change this whole post exists to avoid.

7. Open questions

1. The ten dead methods — delete, or wire up? Deleting makes the published surface smaller and honest. Wiring them up realises the original intent (a virtual folder choosing its own info-dialog content and search strategy). Half a day versus several days, and it changes what ③ publishes.
2. Capability declaration shape — a set of names (warn-and-ignore on unknown, matching EVENT_HOOKS), or a frozen dataclass (typos caught at construction, fields discoverable)?
3. Where the sample lives. xefm/tools/ is for end-user external programs, which this is not.
4. Confirming the registry contract stays "URI string in, PathImpl out", with parsing inside the class — that is what keeps ② testable with ①a nowhere in sight.

[crftwr]

All four tracks are on main, in the order below. 3038 tests pass.

	PR
③	#448	Capabilities become a declaration, not twelve questions
②	#449	A base class worth inheriting from
①a	#450	The scheme list was in four places, and they disagreed
①b	#451	A config can define a place

The open questions, as decided

1. The ten dead methods — delete. Verified against the code first: they have exactly one caller each, the façade delegation, and nothing else. The six flags became a CAPABILITIES set, get_search_strategy and get_display_prefix became SEARCH_STRATEGY / DISPLAY_PREFIX, and the eight façade methods are gone — 387 lines. get_display_title and get_extended_metadata stayed methods, since they compute per instance, but are no longer abstract: their new defaults are what three of the four backends were already returning by hand.

Wiring them up instead would have realised the original intent, but it meant changing live UI behaviour for S3/SSH/archive panes inside the change that publishes the interface. That seemed like the wrong two things to do at once.

2. A set of names, per the post's own lean. supports() takes a string, so a caller written against a later vocabulary gets False rather than an AttributeError, and unknown names are warned about and dropped at class-definition time the way EVENT_HOOKS treats an unknown event.

3. The sample lives in doc/VIRTUAL_FOLDERS_FEATURE.md, inline — a full reg:// registry browser built on winreg. No new directory. The doc's first sample was run verbatim to check it works.

4. Confirmed: URI string in, PathImpl out, parsing inside the class. That is what kept ② testable with ①a nowhere in sight, and it needed one small addition to hold — see below.

Two changes to the plan

③ landed before ②. Writing ReadOnlyPathImpl first would have meant giving it twelve capability methods that ③ then deleted. The ordering constraint §6 actually argues for — both before ①b — still holds.

Path._from_impl was added to path.py, which §6 had ② touching only a new module. The arithmetic has to return a Path, and every backend builds one with Path(uri) — which asks the scheme factory which class to build. A backend the factory has never heard of would find its own parent coming back as a local path. Six lines, and it is what makes §7.4's contract true for an outside class rather than just for the three inside.

The measurement

MockPathImpl was 384 lines and 61 methods. On UriPathImpl it is 85 lines and 15 (writable); the read-only backend beside it is 25 lines and 5. ReadOnlyPathImpl asks for exists, is_dir, iterdir, stat, open and nothing else.

test_published_base.py asserts that abstract set is exactly those five names, so a new @abstractmethod reaching PathImpl without a default here fails the suite rather than a user's config. That is the §4 test.

Three things found on the way

scp:// and ftp:// needed a backend, not a deletion. Dropping them from the one list would have made a subshell try to cd into ftp:/h/p. They are registered to UnsupportedPathImpl instead — five methods, no state, the smallest backend in the repo and the in-tree proof that the published base is enough. The URI now survives (ftp://h/p, not ftp:/h/p), is_remote() still says True, exists() says False so a probe reports a missing path, and reading one raises ENOSYS naming the scheme.

The registry had a shadowing bug, found by ①b's override case: register() did not remember what it covered, so overriding s3:// and then reloading the config deleted S3 rather than restoring it. Fixed in #450, where the module lives.

Dogfooding #451's own sample found three more. The template carried the class and the PATH_SCHEMES = {...} line in one commented lump at class indentation, so uncommenting it in place put the class inside class Config: — where a class body is not a scope its methods can see, and NOTES comes back as NameError on the first call. Registration cannot catch that: all five methods are present, nothing is abstract, nothing warns.

Worse, nothing reported it. Jump to Path's validate called exists() straight through InputDialog._accept, unguarded, so the NameError left through the dialog's key handler: Enter did nothing and nothing said why. It now reports the traceback — naming the line in the user's config.py — and shows the exception in the dialog. run_guarded could not be reused as it stood, because these crossings need the answer and not just the call, so the reporting half is split out as report_user_failure.

(The third was mine: the docs said Jump to Path is Ctrl-J. It is Shift-J.)

Not in it

• Writable virtual folders as a promised surface. PATH_SCHEMES is designed around ReadOnlyPathImpl; nothing stops a config registering a writable UriPathImpl subclass, but the nine write operations are then its own problem.
• The absorption of s3.py's and ssh.py's ~290 lines of duplicated arithmetic into UriPathImpl. The class exists for it now; neither backend has been moved onto it.
• get_extended_metadata still has no caller. It kept a default instead of being deleted, so wiring up a virtual-folder-specific details panel is additive whenever someone wants one.

user_api.API_VERSION stays 0, and a config that defines a scheme gets the existing preview notice, now counting N path scheme(s).