Skip to content

Commit 310e278

Browse files
pi0claude
andcommitted
perf(unstorage): stop downloading objects to discard them, and walk in one round trip
`open(path, "w")` is the most common write a mount ever sees — every shell `>`, every `tar -x`, every editor save — and `acquire()` read the object unconditionally, so overwriting a 1 GB object transferred 2 GB. It now passes the empty value the `O_CREAT` branch already had a mechanism for; `truncate(p, 0)` likewise. An already-open path still shares its one buffer, so a truncate through one handle is seen by the others exactly as before. The path walk did a point lookup *and* a full prefix listing per ancestor, so a `stat` of a missing `/a/b/c/d/e` cost ten round trips, five of them listings. It is now one point lookup per component issued together, then a single listing at the deepest — equivalent because a missing component can never have a key below it, so a shadowing component always precedes a missing one. Ten round trips become seven, listings five become two. A per-call scope memoizes both kinds so `readdir`, `rmdir`, `rename`'s emptiness checks and its directory-move loop reuse the listing the walk already paid for. The memo lives for one driver method call and is then dropped; nothing here reads a key back after mutating it, so it cannot serve a stale answer. Where it costs more, it is always point lookups and never listings: the shadow check no longer short-circuits, so a path whose first component is a key costs *d* lookups rather than one. That is the trade — one round trip of depth for every path that resolves, which is nearly all of them. `maxDepth` is **not** used, and not for the reason the review assumed. unstorage forwards the option to every driver but only `fs`/`fs-lite` implement it, and the two disagree on what depth means — `filterKeyByDepth` counts separators in the absolute key, `readdirRecursive` counts levels below the base — so `maxDepth: 1` on a base of depth 3 returns nothing on an S3-backed store and everything relevant on an fs-backed one. It would have been a correctness bug, not a no-op. No listing got cheaper; `getKeys(prefix)` still enumerates the subtree. Every saving here is a removed round trip. Also guards the zero-length read at the `buffer.set`, matching `memory` — with `validateRange` now correctly returning rather than throwing for that case, an unguarded `set` threw a bare `RangeError` that `errnoOf` mapped to `EIO`. Refs #1 (group I). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent e80c8ff commit 310e278

3 files changed

Lines changed: 441 additions & 54 deletions

File tree

docs/1.guide/3.drivers/1.built-in.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,15 @@ Three characters cannot survive the round trip and are refused with `EINVAL` rat
111111
- **Permissions and timestamps are an overlay**, held in memory for the life of the driver and seeded from `getMeta`. `chmod`, `chown` and `utimes` therefore work — `cp -p` and `tar -x` need them — but a store that cannot hold that metadata does not gain the ability to.
112112
- **A key that is also a prefix** (both `a` and `a:b` exist) has no tree that represents it. The key wins: `a` is a file, and `a/b` is `ENOTDIR`.
113113

114-
Worth knowing before mounting something remote: `readdir` lists every key under the directory's prefix, because the `Storage` interface has no shallow listing, and `stat` falls back to fetching a value to measure it when the underlying driver's `getMeta` reports no `size`.
114+
### Costs worth knowing before mounting something remote
115+
116+
Every one of these is a network round trip, so what matters is how many of them an operation makes — not how long each takes.
117+
118+
- **A listing is the only existence test a directory has.** A directory is a key prefix and has no key of its own, so "does `/a` exist" is `getKeys("a")` — on S3, a paginated `ListObjectsV2` over the whole subtree to answer a boolean. `readdir` is the same call for the same reason: the `Storage` interface has no shallow listing to ask for. `GetKeysOptions.maxDepth` does not help, and it is worth knowing why: only the `fs` and `fs-lite` drivers implement it, so for every other driver unstorage applies it as a client-side filter over the full listing it already fetched — and the two readings do not even agree on what a depth is (`Storage.getKeys` counts separators in the absolute key; `fs` counts levels below the base). There is no "stop after one" in the interface at all.
119+
- **Nothing is asked twice within one call.** One listing is reused for every question a single operation asks about the same prefix — `readdir` classifying a directory and then listing it, `rmdir` and `rename` classifying a destination and then asking whether it is empty. The reuse ends when the call does: the store is shared, so an answer kept any longer would be a stale one.
120+
- **Resolving `/a/b/c` costs a point lookup per component**, which is what catches a store holding both `a` and `a:b`. They are issued together rather than one level at a time, so the walk is one round trip deep rather than as deep as the path, and it needs exactly one listing for the whole walk regardless of depth.
121+
- **`stat` of a file falls back to fetching the value** to measure it, when the underlying driver's `getMeta` reports no `size`.
122+
- **`O_TRUNC` does not fetch what it is about to discard.** `open(path, "w")` and `truncate(path, 0)` write over the object without reading it first — otherwise overwriting a 1 GB object would move 2 GB. Any other truncation length keeps a prefix of the value and still has to read it.
115123

116124
## Which to use
117125

0 commit comments

Comments
 (0)