filetx

Transactional filesystem operations — batch moves, copies and deletes that either all commit or all roll back.

Stack: Python 3.10+, standard library only, zero runtime dependencies.

pip install filetx

from filetx import Transaction

with Transaction(journal="release.jsonl") as tx:
    tx.mkdir("dist", exist_ok=True)
    tx.move("build/app.bin", "dist/app.bin")
    tx.copy("build/manifest.json", "dist/manifest.json")
    tx.delete("build", missing_ok=True)

Nothing happens until the block exits. If any step fails, every completed step is undone. If the process is killed part-way through, filetx undo release.jsonl finishes the rollback from the journal it left behind.

---

1. The problem

Multi-step filesystem work has no rollback. A release script that creates a directory, moves three artefacts into it and deletes the build tree has four chances to fail, and if it fails on the third the disk is left in a state that matches neither the before nor the after. The usual responses are to copy everything to a backup directory first — which costs time proportional to the data, so nobody does it for large trees — or to write bespoke cleanup code per script, which is only exercised on the day it is needed and is therefore usually wrong.

Python has shutil for individual operations and atomicwrites for making a single file write atomic. Neither gives you a batch that is all-or-nothing. filetx is that missing piece: record the operations, commit them, and get the tree back unchanged if anything goes wrong — including if the process dies.

This started as the transaction log inside a file-organiser tool I built, where a half-finished reorganisation of someone's photo library is a genuinely bad outcome. The engine turned out to be the interesting part, so it was extracted, generalised and published.

2. What it operates on

There is no dataset here — this is a library, and its input is whatever tree the caller points it at.

Input	Arbitrary files, directories and symlinks on a local filesystem
Scale tested	Trees up to 5,000 files; single renames are size-independent
Benchmark data	Synthetic, generated by scripts/benchmark.py (seed 42), not committed
Package licence	MIT
Dependencies	None at runtime. pytest, pytest-cov and ruff for development

Non-goals. Not a distributed transaction manager, not crash-safe against media failure, and not safe against another process mutating the same paths concurrently — see Limitations.

3. Architecture

flowchart TD
    A["tx.move / copy / delete / mkdir<br/>(recorded, not performed)"] --> B["commit()"]
    B --> C["plan_undo — validate,<br/>compute undo record"]
    C -->|invalid| R
    C --> D["journal — write undo record, fsync"]
    D --> E["apply — change the filesystem"]
    E -->|OSError| R
    E --> F{"more operations?"}
    F -->|yes| C
    F -->|no| G["journal — commit"]
    G --> H["purge staged material"]
    R["revert applied operations,<br/>in reverse order"] --> S{"all reverted?"}
    S -->|yes| T["journal — rollback<br/>raise TransactionError"]
    S -->|no| U["journal left unsettled<br/>raise RollbackError"]
    U -.->|"later: filetx undo"| R

Loading

The undo record is written to the journal before the change it describes is attempted. That is the whole design: a record written afterwards is useless to a process that died in between.

Writing it first requires knowing, in advance, where a file is about to be moved to — so staged names are derived deterministically from (path, transaction id, plan index) rather than allocated at random.

The remaining ambiguity is that the journal can describe a change that never actually happened. Every revert therefore inspects the real filesystem before acting instead of trusting the record, which makes reverting idempotent: running it twice, or against an operation that never applied, does nothing.

4. Key decisions & tradeoffs

Decision	Chose	Over	Why
Making deletes reversible	Rename the target to a hidden sibling in its own parent directory	Copy it to a staging/backup directory	A sibling is guaranteed to be on the same filesystem, so it is one atomic rename regardless of size — roughly three orders of magnitude faster on a 5,000-file tree (§5). It also nests correctly: deleting a/b.txt then a stages the file inside a, then renames a wholesale, and reverse-order rollback restores a before looking for b.txt inside it.
When to compute the undo record	Before applying (write-ahead)	After applying, returning what was done	The "after" version cannot recover a process killed between the change and the log write — precisely the window that matters. Cost: staged names must be deterministic, and the journal may record changes that never happened, which is why every revert is state-checking.
Operations execute	Deferred — recorded, applied on commit()	Eagerly, as each method is called	Makes the plan inspectable and gives a dry run (tx.describe()) for free, and an exception in the caller's own code inside the with block costs nothing to undo.
Failure signalling	TransactionError and RollbackError as distinct types	One exception for "the commit failed"	"Your change didn't happen and the tree is fine" and "your change didn't happen and the tree is in an unknown state" demand completely different responses. Collapsing them would be the most dangerous thing this library could do.
Recovery vs. leftover staged data	Report it, never delete it	Clean up automatically	If a revert declined to restore something because the original path is occupied again, the staged copy may be the only copy left. Unattended deletion there is the one unrecoverable mistake available; filetx undo prints the paths and stops.
Cross-filesystem moves	Detect EXDEV, degrade to copy-then-stage	Refuse, or always copy	Keeps the fast path fast and the slow path correct. Documented as O(size) rather than O(1). Any other OSError propagates — a permission error must not be silently treated as a volume boundary.
Journal format	JSON Lines	SQLite, or a binary log	An operator looking at a half-finished batch can read it with cat. Appending one short line is atomic enough at this size, and a torn final line is detected and tolerated.

5. Results

Measured on Windows 11 (NTFS), Python 3.13.9, 12-core CPU / 64 GB RAM, median of 3 runs. Reproduce with uv run python scripts/benchmark.py --files 5000 --size 1024. Run-to-run variation on this machine is roughly ±15%, and the speedup ratio has ranged 1,000–1,300× across runs — the figure worth trusting is the order of magnitude, not the exact multiple.

Metric	Value	Notes
Make a 5,000-file delete reversible, then undo it	8.8 ms	Two renames
Same outcome via copy-to-backup then restore	9,178 ms	shutil.copytree + rmtree + copytree
Speedup on the reversibility path	~1,000×	Widens with tree size; staging is size-independent
Commit the same delete (data actually removed)	1,200 ms	Staging is O(1), deleting is not — stated so the number above is not mistaken for magic
Tests	73 passing, 100% line coverage	Includes two tests that os._exit() a real subprocess mid-commit, on either side of the rename, then recover from the journal
Python versions	3.10, 3.11, 3.12, 3.13	All four run in CI, not just claimed in metadata
Platforms	Linux, Windows, macOS	Windows and macOS on 3.13; this library is mostly os.rename, which is where platforms disagree
Journal write, fsync=True (default)	1.57 ms/record	0.01 ms with fsync=False — this is the scaling ceiling, see §7
Runtime dependencies	0

The benchmark tree is synthetic and generated by the script above. The comparison is against shutil, i.e. what you would write by hand, not against another library.

6. How to run

git clone https://github.com/Prithv122/filetx.git
cd filetx
uv sync
uv run pytest --cov=src --cov-report=term-missing

No environment variables, services or datasets are needed. Other useful commands:

uv run python scripts/benchmark.py     # reproduce the numbers in section 5
uv run ruff check . && uv run ruff format --check .
uv build && uv run --with twine twine check --strict dist/*

Recovering an interrupted run

filetx inspect release.jsonl   # what was this transaction doing, and how far did it get?
filetx undo release.jsonl      # put it back

inspect marks each operation done, PARTIAL or -. undo rolls an unsettled transaction back, cleans up after a committed one, and reports — but never deletes — staged data whose original path has since been reoccupied.

Limitations

• No locking. Two transactions touching the same paths concurrently will interfere. Serialise them yourself.
• fsync durability only. The journal is fsync-ed before each change, so a process kill or power loss is recoverable, but a lying disk cache or media failure is not. Pass fsync=False to trade that away for speed.
• Staged material is visible. During a transaction, .filetx-* entries exist alongside your files. A crash leaves them until filetx undo runs; filetx.STAGE_PREFIX is exported so scanners can skip them.
• Cross-filesystem moves are O(size). They fall back to copying.

7. What I'd change at 100× scale

The design assumes a batch you can hold in memory and a journal you read whole. At 100× — hundreds of thousands of operations per transaction — three things break, in this order:

1. The plan is written to the journal as one record. A single JSON line holding 500k operations has to be serialised and fsync-ed before the first change happens, and re-parsed in full during recovery. I would stream the plan as one record per operation and make read_journal incremental.
2. fsync per operation dominates. Measured at 1.57 ms per record here (versus 0.01 ms unflushed), so 500k operations is about 13 minutes of waiting on the disk before any useful work happens. I would batch the flush — group commits of N records — accepting a bounded window where the journal lags reality, and handle it by making recovery re-verify the tail against the filesystem, which the state-checking reverts already support.
3. Rollback is serial. Undoing 500k renames one at a time wastes an SSD's parallelism. Reverts of operations on disjoint paths could run in a thread pool; the ordering constraint is only between operations whose paths nest, which is a partial order the planner could compute up front.

What I would not change is the staging strategy — it is the part that gets better with scale, not worse.

---

References

The write-ahead ordering and the do/undo/purge split follow standard database recovery practice; ARIES (Mohan et al., 1992) is the canonical description, and the naming of the three phases here is deliberately borrowed from it. No implementation was consulted — the filesystem constraints are different enough that the resemblance is conceptual.

Licence

MIT — see LICENSE.