More archive formats — a handler registry, with 7z as libarchive's first demonstration

[crftwr]

Prompted by #376, which I opened after hitting the gap myself: I wanted to open
a .7z and could not. That is the concrete demand, and I would rather not
dress one format up as a survey of formats.

But I do not want a 7z-shaped fix. The next person will arrive with .lha or
.rar or .iso, and the useful question is what they have to do at that
point. So the plan is a registry underneath, libarchive as the engine on top of
it, and 7z as libarchive's first demonstration rather than as a
special case solved its own way.

This follows the same shape as #378: the format is the visible ask, the
mechanism underneath is the thing worth getting right, and here it is cheaper
than expected because most of the abstraction already exists.

py7zr is deliberately not part of this. It would close #376 with no native
code at all, but it would also mean two 7z implementations, two solid-block
behaviours, two password models, and a demonstration that never gets
exercised on the maintainer's own machine. One engine.

1. How it works today

Better than #378 led me to believe. ArchiveHandler (xefm/archive.py:344) is
already the contract that discussion sketched:

open() / close()
list_entries(internal_path) -> List[ArchiveEntry]
get_entry_info(internal_path) -> ArchiveEntry | None
extract_to_bytes(internal_path) -> bytes
extract_to_file(internal_path, target_path)

ZipHandler (:472) and TarHandler (:833) already sit on it, and
ArchiveEntry and ArchiveCache (:1160) are format-agnostic. A new format
does not need any of that reshaped.

Three places need work, and only the third is real surgery.

The dispatch — ArchiveCache._create_handler, archive.py:1268-1283. An
if/elif chain on filename.endswith(...), raising ArchiveFormatError
otherwise. This becomes a table lookup. One subtlety to carry over: the chain
depends on .tar.gz being tested before .tar, so longest-suffix-first has
to become part of the registry contract rather than an artifact of source
order. app.py:4673 already gets this right explicitly, with a comment saying
so.

The create side — _ARCHIVE_EXTS and _TAR_MODES, app.py:4673-4681. The
"second place" from #378, but it turns out to serve a different purpose:
P (create), U (extract) and _archive_basename, driving tarfile /
zipfile writes directly. It is not browse-side dispatch at all. Reading and
writing want different registries, or one registry where writing is optional.

Encryption — the actual leak. archive.py:2133 and :2159 both do
isinstance(handler, ZipHandler), and zip_encryption_status() (:309),
verify_zip_password() (:326) and zip_encryption_status_path() (:2171)
are module-level ZIP functions that bypass the ABC entirely. 7z is routinely
encrypted, so encryption_status() / set_password() have to be lifted onto
ArchiveHandler before the new handler can deal with a password-protected
archive. This is the only place where existing ZIP behaviour genuinely gets
moved.

2. Why the registry earns its place regardless

It collapses format knowledge from an if/elif chain plus two isinstance checks
into one table, which is a tidy-up worth doing with or without new formats.

It is also the cheapest possible answer to "what does the next person do".
Today a new format means patching _create_handler and adding an isinstance
branch; with the registry it means writing a class against a contract that
already has three customers. Three is the number that matters: zip and tar
have been exercising this ABC all along, and 7z arriving as the third is what
validates the contract without having to guess at hypothetical formats. A
registry designed around one new format would be a registry designed around
that format.

And it is the retreat path. If libarchive turns out to be the wrong bet,
swapping in a pure-Python 7z engine is a one-line change to a table rather than
an unpicking.

3. Why 7z is the right thing to demonstrate it with

Every property of libarchive that could go wrong gets exercised by a format I
open regularly: the capability probe, the loader search order, GIL release on a
worker thread, password-protected entries, and the cost of per-entry extraction
inside a solid block. A demonstration built on .lha would be a demonstration
nobody notices is broken.

Once it works, rar / lha / cab / iso / xar arrive as table entries rather than
as projects, because the format readers for all of them are libarchive's own
implementations — external libraries are only needed for the compression
layer. RAR in particular comes without the non-free unrar binary that
rarfile requires. And being ctypes, calls release the GIL, which matters for
a worker-thread listing path.

4. What libarchive costs

Codec support is fixed at build time. ENABLE_ZLIB / ENABLE_LZMA /
ENABLE_ZSTD / ENABLE_LZ4 / ENABLE_BZip2 (CMakeLists.txt:223-234) feed
HAVE_* macros that switch #ifdefs. There is no runtime plugin mechanism.
Static vs. shared linking of the codecs is an ordinary linker choice on top of
that.

Static linking is cheaper than expected. The README states that static link
pollution is minimised: a compression that is never explicitly enabled is not
pulled into a statically linked binary, and does not require linking the
corresponding library.

The Windows dependency tree is shorter than expected. ENABLE_CNG uses
bcrypt.h for hashes instead of OpenSSL, and ENABLE_WIN32_XMLLITE covers
xar's XML instead of libxml2/expat. What is left to bundle is zlib, bzip2,
liblzma, zstd and lz4 — and zstd and lz4 can be dropped outright if the target
list is zip / tar.* / 7z / rar / lha / cab / iso.

The silent fallback is a trap. With a codec absent, libarchive does not
fail — it spawns an external program.
archive_read_support_filter_gzip.c:263 calls __archive_read_program(f, "gzip -d") and logs "Using external gzip program"; xz, lzma and lzip do the
same at archive_read_support_filter_xz.c:755-783. A process per entry, and on
Windows those binaries do not exist.

Random access is not supported. The README says so directly, noting that
libarchive can sometimes re-open and rescan from the beginning fast enough to
compensate. For virtual-directory browsing this lands on extract_to_bytes(),
not open() — building the tree is one pass either way, which the
open()-caches-the-structure contract actually suits. Solid 7z stays O(n) per
entry regardless, and that cost needs measuring before this ships.

5. Nobody ships the binary, so we build it: xefm-bin-deps

libarchive-c 5.3 on PyPI is py3-none-any — pure ctypes, no binary. The
shared library has to come from somewhere, and it does not exist as a
maintained, redistributable artifact:

Source	State
libarchive upstream	Windows binaries stopped after libarchive-v3.5.3-win64.zip; 3.6.0 onward is source-only (current 3.8.9). No macOS binary, ever
conda-forge libarchive	3.8.5, all platforms, actively updated — but linked against conda-forge's own libzstd / liblzma / libxml2 / openssl with conda rpaths. Not extractable
ShiftMediaProject/libarchive	Exactly the right shape (prebuilt static and shared, 32/64-bit) — last release v3.4.0
hermeticbuild/bsdtar-prebuilt	Statically linked, explicitly a workaround for upstream not shipping binaries — but it is the CLI, not the library
Homebrew / MSYS2 / vcpkg	Package-manager-local; brew dylibs carry Homebrew paths, MSYS2 drags mingw runtime DLLs

Upstream dropping binaries and ShiftMediaProject going stale is itself evidence
that maintaining this artifact is unrewarding. The one reusable piece is the
conda-forge feedstock recipe (BSD-3) — the build configuration, not the
output.

So: a separate repository, GitHub-only, no PyPI. Not vendored into this tree —
the CVE cadence of zlib / bzip2 / liblzma has to be independent of XeFM's
release cadence, and make venv should not grow a CMake toolchain. Same
reasoning that keeps PuiKit separate.

Named for the role rather than the library, because libarchive is unlikely to
be the only native dependency. If #332 lands on C/Migemo, it needs the
identical machinery: per-platform prebuilt shared library, macOS signing,
download at build time. python/cpython-bin-deps is the precedent for the
name.

It has to be a user-facing repository, not just build material. Terminal
users on Windows and macOS will be pointed at its releases by name, so it needs
per-platform asset names, install instructions, a Gatekeeper note, and guidance
on where to put the file and how to set LIBARCHIVE. The README should still
say plainly that it exists to serve XeFM and carries no support promise for
other consumers — but it cannot read like an internal artifact.

Publishing it does not make us a binary distributor for anyone else. It does
make us responsible for rebuilding when a codec CVE lands, which is roughly a
once-or-twice-a-year event across the five libraries involved. The real cost is
not the rebuild but the re-shipping: notarization, and Store certification.

6. Three supply paths, one loader

• Desktop bundles download a pinned, checksummed xefm-bin-deps asset at
  build time. Not "latest" — a Store submission has to be reproducible. The
  bundling step must re-sign the dylib with our Developer ID, which is what
  keeps macOS library validation satisfied without a
  disable-library-validation entitlement.
• Terminal users use whatever their system provides, or download the same
  asset by hand, or build their own. On Linux the system copy is usually
  present and sufficient.
• Nobody is required to have it. Absent libarchive, the registry has fewer
  handlers in it and zip / tar keep working. 7z not opening is a missing
  feature, not a broken install.

Search order, documented: bundled → LIBARCHIVE environment variable (which
libarchive-c already honours) → find_library("archive").

Registration must be driven by a capability probe, not a version check,
because on two of the three paths we do not control how the library was built.
archive_version_details() (archive.h:191) returns only the codecs actually
compiled in, with their versions — its implementation is switched on the same
HAVE_ZLIB_H / HAVE_LZMA_H / HAVE_BZLIB_H / HAVE_LZ4_H / HAVE_ZSTD_H
macros. Register the suffixes that string justifies and no others. This is also
how we avoid ever hitting the external-program fallback silently.

Which library was loaded, and what it can do, should be logged at startup. With
three supply paths a bug report has to carry that automatically — #360 already
makes the log pane copyable.

7. What terminal users have to do

More than nothing, and that is accepted. Anyone who reached XeFM through
pipx install has already chosen a tooling-literate path, and 7z support is
worth one documented step. macOS's own libarchive is one upstream calls
possibly too old to work properly, so the likely instruction on macOS and
Windows is "download the asset from xefm-bin-deps".

Two consequences for the docs:

• The archive section of the User Guide needs an "if 7z does not appear" path,
  and it should name the log line from §6 as the diagnostic.
• The supported-format list stops being static. It now depends on what the
  loaded library reports, so anything enumerating formats has to be generated
  from the registry rather than written down — the same class of drift doc/HELP_DIALOG_FEATURE.md's section list has drifted from _HELP_SECTIONS #389 is
  about, and worth not creating deliberately.

FILE_ASSOCIATIONS (xefm/_config.py) remains the zero-setup fallback: it can
hand *.7z to an external tool today. No in-archive browsing, but it is the
honest answer for someone who does not want to install anything.

8. Two decisions to settle first

The threading contract. Same point as #378, and archive listing is the
sharpest case of it: "a registry implementation may be called from a worker
thread and must not touch the UI" is a promise that cannot be added after the
first registry ships.

Read-only registry, or read/write. Reading is where the demand is, and RAR
cannot be written at all. My inclination is a read-only ARCHIVE_HANDLERS,
leaving _TAR_MODES and the create path alone. Two tables survive, but they
mean different things, which is honest. Note this leaves P unable to create
7z archives — a gap worth stating out loud rather than discovering later.

Whether any of this is exposed through user_api.py is a separate call and can
wait.

9. Stages

Stage	Content	Effort
1	ARCHIVE_HANDLERS registry; lift password handling onto the ABC; longest-suffix matching in the contract. No new formats	~half a day
2	xefm-bin-deps: libarchive build per platform, curated codecs, release assets, install docs	1–2 days, plus a Windows machine
3	libarchive handler + capability probe + loader search order + startup logging. 7z appears	~1 day
4	Bundle into the desktop builds: pinned download, checksum, macOS re-signing	~half a day
5	rar / lha / cab / iso / xar as table entries, once 7z has proven the path	small

Stage 1 stands alone and can land first. Stage 2 is the long pole and the one
that needs hardware.

Open questions

1. What do the system libarchive builds on macOS and Windows actually report
   from archive_version_details()? This decides how often the "download the
   asset" instruction is the answer rather than the exception.
2. Measured cost of extract_to_bytes() on a solid 7z of realistic size — the
   number that decides whether per-entry extraction needs a caching strategy
   before shipping rather than after.
3. Read-only registry, or one registry with optional write capability?
4. Which codecs go into the xefm-bin-deps build — dropping zstd and lz4
   halves the CVE surface, at the cost of zstd-compressed 7z entries.

[crftwr]

All five stages have landed.

Stage
1	ARCHIVE_HANDLERS, password handling lifted onto the ABC, longest-suffix in the contract	#397
2	xefm-bin-deps — libarchive-3.8.9-2, Windows only; macOS and Linux use their system libraries	#398
3	The libarchive handler, capability probe, loader order	#397
4	Bundled into the Windows builds, pinned by tag and SHA-256	#398
5	.rar, .iso, .cab, .cpio, .rpm — table entries, as predicted — plus .tar.zst from the standard library	#399

The open questions, answered.

1. What do the system builds report? macOS 3.7.4 with zlib/liblzma/bz2lib — enough for every format XeFM offers, so "download the asset" is the Windows instruction and not the macOS one. Linux distributions carry more (libzstd, liblz4, openssl). The formats registered are the same six on all three.
2. Cost of extract_to_bytes() on a solid 7z? Sidestepped rather than measured. iter_extract() extracts in one forward pass, so the quadratic case never arises for U; browsing still pays O(n) per entry and that number is still unmeasured.
3. Read-only registry, or read/write? Read/write, but not as one table. ArchiveFormat.writer is set only where the writer arrives with the engine; zip and tar stay with XeFMApp, and _writable_formats() is the union. RAR, CAB and RPM being readable and not creatable is what the split is for.
4. Which codecs in the build? zlib, bzip2, liblzma and libzstd, with CNG replacing OpenSSL and XmlLite replacing libxml2.

Three things the discussion got wrong, all found by building it.

• Encrypted 7z is not a build-flag question. A 3.8.9 build linked against CNG refuses the same archive macOSs crypto-less 3.7.4 does: through 3.8.9, zip is the only format libarchive decrypts. can_decrypt_7z() stays because it is the difference between saying "not supported" and rejecting every password typed.
• The external-program fallback is a filter mechanism, not a format one. A zstd-compressed entry inside a 7z fails cleanly with ZSTD codec is unsupported; only a .tar.zst spawns zstd -d. §6 was right about the trap and it was caught in the act — macOSs libarchive was shelling out to Homebrews zstd for .tar.zst, which is why that format goes through Python 3.14s tarfile instead. Of the registered formats only .rpm can still reach the fallback.
• ctypes.util.find_library is not a reliable way to find a library. On musl it returns None with libarchive installed at /usr/lib/libarchive.so.13, -dev symlink or not, so XeFM offered zip and tar and nothing else on Alpine. It names the soname itself now (Test XeFM on Linux in Docker, and fix what that found #400).

Not done, deliberately. No format can be created with a password — libarchives 7z writer has no encryption. xar is out because libarchive 3.8.9 over-reads its entries (a 7-byte member comes back as 13, NUL-padded), and lha because the same library fails fatally on a Shift_JIS filename while reading an ASCII one beside it — the wrong half to support for the archives that format is found in. Both reasons are recorded in _CANDIDATES so they are not re-derived.