Robustly wrapping stores that have key- and value-taking methods

[thorwhalen]

Design rationale, consolidated so the next person does not re-derive it. Not a work item —
the work items are #83, #82, #18, #10, #6, #5, and PRs #84 / #85.

dol's core promise is that you can wrap a store with layers of transformations and keep using
it as a Mapping. That promise holds for the Mapping protocol and fails for everything else:
a wrapped store's non-dunder methods receive the outer, unmapped key, and its methods'
internal self[k] bypasses the transforms. Those are the two halves of one root cause, and a
13-package census says the family has quietly been living with both.

This writes down what we know, what we tried, and what each option actually costs — including
three options that look right and are not.

---

1. The mechanism, precisely

dol wraps by delegation (has-a): wrap_kvs(SomeClass) returns a Store subclass holding
an instance of SomeClass in self.store. The transform hooks
(_id_of_key / _key_of_id / _data_of_obj / _obj_of_data) live on the wrapper. The inner
instance never sees them.

Two consequences, in opposite directions:

	direction	issue
A method receives an unmapped key	outside → in	#83
A method's self[k] bypasses the transforms	inside → out	#18

And two delegation routes, which matters because a fix covering one is a silent no-op on the
other:

route	when	site
Store.__getattr__	instance-wraps, and mk_relative_path_store subclasses	dol/base.py:742
DelegatedAttribute.__get__	class-wraps; delegate_to installs one descriptor per attr of dir(wrapped)	dol/base.py:279, installed dol/base.py:416-480

Both return the method bound to the leaf.

Nothing raises. Capability detection cannot see it either — a @runtime_checkable Protocol
checks method presence, and since 3.12 isinstance uses getattr_static, so
isinstance(w, SupportsUrlFor) is True for a class-wrap and False for an instance-wrap. In
neither case does it say anything about whether the key is right.

---

2. Scale — the census

13 sibling packages surveyed, every claim re-verified by source read plus a runnable repro.
Index and per-package issues in #83.

The defect is overwhelmingly latent: it bites when a user applies a key codec, and most of
these packages never wrap their own stores. 12 survey claims were refuted outright. Being
accurate about this matters — an earlier draft of the s3dol write-up said "already destroying
data in production", and the evidence does not support that.

What it does support:

• focal — confirmed-live throughout, inherited from dol.filesys.Files.
• dol itself — Files(d).is_valid_key(k) is False for a key the store yields, breaking
  all(s.is_valid_key(k) for k in s). Fixed in fix: two delegation bugs in dol's own code (content_url, filesys key validation) #85.
• cosmodol.CosmosItems.replace — under a key codec, silently overwrites a different,
  real document in full.
• pydrivedol.GDReader.get_url — returns a URL for the wrong file and grants
  anyone/reader permission on it.
• xdol, unbox, lexis — the wrap_kvs will wrap the instance but self of instance is not wrapped #18 direction, already migrated to wrapped_self.

The shape that stays clean is instructive: azuredol's ContainerStore has essentially no
key-taking public methods. Its rich per-object surface lives on BlobHandle, which binds its
blob at construction, so a key codec over the store cannot corrupt it. It is not safe because of
where its prefix lives; it is safe because it has almost no seam to get wrong.

---

3. The option space

A — document it, use wrapped_self

Ships today, and it is what #18's design doc recommends for Phase 1.

But wrapped_self has its own silent failure mode. A delegated bound method holds no
reference to the wrapper, so when nothing else does, the wrapper is freed before the body runs
and _register_wrapper_backref's cleanup removes the registry entry — indistinguishable
from "never wrapped":

s = KeyCodecs.prefixed('x/')(BR('logs/'))
s.m_abs('b.txt')                                    # 'logs/x/b.txt'  correct
KeyCodecs.prefixed('x/')(BR('logs/')).m_abs('b.txt')
                                                    # 'logs/b.txt'    WRONG, silently

Where the leaf owns a prefix, the wrong answer is a plausible str, so a type check cannot
catch it. The predicate is "no live strong reference", not "temporary" — a temporary in a
reference cycle silently starts working, so the failure is intermittent. Reproduced on
CPython 3.10–3.14. Detail in #18.

Verdict: a real guardrail, and better than nothing — but not a correctness mechanism, and it
should stop being described as one.

B — declarative key-method registration

_key_methods = {'url_for': 0, 'delete_many': ('iter', 0)}, with the wrapper generating mapped
delegates.

dol already has this: wrap_kvs(ingoing_key_methods=…, outcoming_key_methods=…)
(dol/trans.py). It is untested (the TODO says so) and verified broken for leaf-defined
methods on both wrap paths — getattr(store_cls, name) at decoration time finds a
DelegatedAttribute or nothing, and the generated body calls super(store_cls, self).<name>(),
which does not consult Store.__getattr__. It fails loudly, at least. Building on it means
replacing it.

The deeper problem is that a method the author forgets to declare fails the same silent way. The
part that actually holds is not the registry but a reflective test that enumerates public
methods and fails on any undeclared one.

Verdict: the guard is worth more than the registry.

C — free functions

url_for(store, k) instead of store.url_for(k), resolving through the chain with
inner_most_key.

This is dol's own idiom (content_url, get_content, put_content, add_content), and it
sidesteps A's lifetime problem because the caller holds the store.

But it is not categorically safe. inner_most_key walks .store applying each layer's
_id_of_key, which breaks when a layer is not a dol Store:

chain	wire key	free function
any dol-shipped wrapper	—	correct
hand-rolled __getattr__-passthrough delegator	logs/b.txt	logs/logs/b.txt
key codec over a plain .store-holding middle layer	logs/x/b.txt	x/b.txt

A passthrough __getattr__ resolves _id_of_key to the leaf's bound method, so the walk
applies it twice; a middle layer without _id_of_key truncates the walk. Every dol-shipped
wrapper is safe — but dol/base.py:514-518 ships a documented hand-rolled Delegator recipe of
exactly the breaking shape.

Note also that dol's own instance of C, content_url, had the bug until #85: it did a flat
getattr(store, 'url_for')(key).

Verdict: more reliable than A, not reliable. Good for operations that are not keyed.

D — rebind delegated methods to the wrapper

Rejected, with running-code evidence, in misc/docs/dol_issue18_design.md §4. The fatal
defect is not blast radius: rebinding binds self to the innermost Wrap, so under a
Pipe stack it does not fix the case it exists to fix — 4 of the 6 sites that survey found
are Pipe stacks — and stacked-codec writes gain a partial-transform corruption surface. Plus
statically-undetectable crashes: a leaf method calling super().__getitem__(k) compiles
super(SomeClass, self) with self now a Wrap → TypeError.

Its Phase-1 instructions say: do not touch DelegatedAttribute.__get__, the delegate_to copy
loop, or the signature graft.

Verdict: do not re-propose without new information.

E — capabilities as parallel Mappings

Two designs get conflated under this name, and only one works.

As a Mapping-valued attribute (store.urls) — broken. A wrapper does not re-wrap such an
attribute:

KeyCodecs.prefixed('a/')(Leaf)({'a/b': 1}).urls    # {'a/b': ...}  INNER keys

The outer store's keys are ['b'], so store.urls[k] KeyErrors for every key it has. This is
#10's territory. (There is a precedent for the propagation half: Store.__init__
(dol/base.py:720-727) copies KeysView/ValuesView/ItemsView up from the leaf and binds
them to the outer store. Hard-coded to three names, but it is propagation plus outer
binding, which is simpler than re-wrapping.)

As a sibling store — correct by construction. Expose the capability as its own KvReader
over the same key space:

class BucketHandles(KvReader):
    def _id_of_key(self, k): return self.prefix + k
    def __getitem__(self, k): return ObjectHandle(self.bucket, self._id_of_key(k))

__getitem__ is the one thing dol maps correctly at every depth, so this needs no
inner_most_key, no wrapped_self, no guard. Verified correct unreferenced, under Pipe,
under cached_keys, under a value codec, and under the non-Store passthrough layer where C
is silently wrong.

Its cost is real: the user must wrap the sibling in parallel with the data store, because
deriving the codec chain onto a sibling is exactly #10.

Verdict: the best answer available today. s3dol adopted it
(ADR-0011).

F — is-a wrapping

Make wrap_kvs(Class) return a real subclass of Class. Then self inside a leaf method is
the wrapper, the transform hooks are on the MRO, and both #83 and #18 disappear — along with #6,
since the __signature__ graft exists only because Wrap is a generic shell rather than a real
subclass.

misc/docs/dol_issue18_design.md recommends this as the terminal direction for 0.4/1.0, with a
staged plan (opt-in mode='isa' first, default flip gated on a green dependents run). It is
x-large, coupled to #5, and its §9 still lists "commit to C as the terminal direction now?" as
an open question for the maintainer.

Verdict: the actual fix. Everything above is what you do until it lands.

---

4. What this implies for adapter authors today

The selection criterion that turned out to matter is not "which is the permanent answer" but
"which is correct on 0.3.x AND harmlessly redundant once is-a lands".

1. Have as few key-taking methods as possible — ideally none. This is the single biggest
   lever, and it is what makes azuredol robust. A capability that binds its key at
   construction (a handle) cannot be corrupted by a key codec at all.
2. Route keyed capabilities through __getitem__ — sibling stores (E-as-store). Correct by
   construction, no primitive.
3. Free functions for the non-keyed rest (C), knowing the non-Store-layer caveat.
4. wrapped_self only as a guardrail (A), never as a correctness argument.
5. Guard it with a reflective test (B's useful half) that fails on any new public method
   taking a key. Note a name heuristic is not enough: it misses delete(name, ...),
   delete_many(keys), batch(operations), sync_to(target) — several of the worst shapes.

---

5. Open questions

1. Is is-a (F) committed as the terminal direction? wrap_kvs will wrap the instance but self of instance is not wrapped #18's doc recommends it; its §9 still
   asks. Everything else is a holding pattern, so the answer changes how much to invest in the
   holding pattern.
2. Should wrapped_self's lost-reference hole be fixed, or left as a known limitation? A
   two-line change to _register_wrapper_backref makes it detectable-and-loud (detail in wrap_kvs will wrap the instance but self of instance is not wrapped #18).
   Cheap, but it hardens a mechanism F would delete.
3. Should dol grow the sibling-store pattern as a first-class thing — a helper that
   re-derives a wrapper chain onto a sibling store, which is Making conditional_data_trans better, and generally; recursively applying wrappers #10 — or stay out of it and let
   adapters ask users to wrap in parallel?
4. Should SupportsUrlFor stay method-shaped? It currently forces every backend to expose a
   keyed method, which is the shape this whole thread is about. A store-shaped or
   function-shaped seam would not.
5. Is there an inverse of inner_most_key? Mapping a key outward (leaf → caller) has no
   helper, and anything returning keys (prefixes, walk, a query) needs one.

References

Issues: #83 (this problem + census) · #18 (root cause, wrapped_self) · #82 (prefix
corruption) · #10 (recursive wrapping) · #6 (signature freeze) · #5 (wrapper class control).
PRs: #84, #85.
In-tree: misc/docs/dol_issue18_design.md, misc/docs/dol_issue10_design.md,
misc/docs/dol_content_metadata_bifurcation.md §2.2.
Downstream: s3dol#14,
s3dol ADR-0011.

[thorwhalen]

Option G — spec-carried boundary codecs on a flat proxy (design study + prototype)

Following up on the maintainer's proposal (2026-08-10, relayed off-thread — putting it on record here): study wrapt's object-proxy design; wrap incoming/outgoing keys and values in all methods per a specification (e.g. a Protocol annotated with KT/VT); generalize beyond KT/VT to arbitrary "types of interest"; accumulate wrap layers in two flat lists (encoders/decoders) and compile them for speed and validation; take care that method-calls-method never double-applies.

This is now evaluated as Option G in this thread's option space:

• Design study: misc/docs/dol_issue86_design.md (survived a 4-lens adversarial panel + an independent code review; corrections folded in and flagged)
• Prototype: dol/_interface_wrap.py (private, experimental, stdlib-only) + 39 tests — landed in Option G design study + prototype: spec-carried boundary codecs on a flat proxy (#86) #88

The short version

1. wrapt: take the lessons, not the goal. Verified on wrapt 2.3.0: its proxy has exactly our wrap_kvs will wrap the instance but self of instance is not wrapped #18 hole (self inside a wrapped method is the wrapped object — the proxy intercepts only the first hop), pickling refuses by default, and its __iter__ mistake is a ten-year cautionary tale about eagerly-defined dunders. Also: the pure-Python proxy measured ~5× faster than the C extension on CPython 3.12 — a dependency-free carrier costs nothing.
2. The flat compiled stack is the load-bearing idea. Re-wrapping extends a list instead of nesting an object, so — within pure-codec stacks — inner_most_key becomes a total fold, the inverse key mapping (§5.4 of the Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 study — previously nonexistent) exists by construction, the leaf is a strong structural reference (§5.6), stacked wraps pickle by construction, and the measured ~420–510 ns/layer delegation tax collapses to one hop (3.6× faster at depth 6). Scope limit, stated plainly: filt_iter (key-set change) and cached_keys (stateful) are not codecs; mixed compositions still nest.
3. The typed spec is Option B, replaced — with B's verdict answered. "The guard is worth more than the registry": in G the guard is the mechanism — three wrap/compile-time refusal layers (undeclared attributes; unannotated params inside spec'd methods; unsupported shapes) — and the registry is derived from annotations, not maintained. The adapter-side reflective conformance test (ADR-0011 D5) stays; the wrap-time guard serves the other desk.
4. The boundary invariant has fine print. "A method correct on the bare leaf stays correct under the stack" holds iff the codec laws hold: decoder total+injective on the leaf's actual keys (our own prefixed/suffixed codecs violate this on out-of-band keys — the ADR-0006 corruption family), encoder/decoder mutual inverses, container args copied at the boundary. No-double-apply on internal self-calls is structural (verified with a counting encoder).
5. Headline finding — F's verdict needs a correction. Is-a wrapping does not fix Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 for backend-direct method bodies (verified): a body like cosmodol.replace that hands its key straight to the backend still receives the outer key under is-a. So "both Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 and wrap_kvs will wrap the instance but self of instance is not wrapped #18 disappear" overclaims; F and G serve disjoint populations (F: outer-domain bodies, wrap_kvs will wrap the instance but self of instance is not wrapped #18; G: leaf-domain bodies, Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83), and this thread's open question 1 ("is is-a committed as the terminal direction?") becomes "committed for which population". Bonus hazard for F's Phase 3: a leaf that owns _id_of_key (s3dol's shape) gets its prefix arithmetic shadowed by a naive is-a hook.
6. A decision rule for the two blessed patterns (proposed): spec expressibility is the line. Shapes the spec can express (scalar KT, Iterable[KT], tuple[KT, VT]…) MAY be methods iff spec'd; shapes it refuses (key-space-shifting returns, cross-keyspace values, keys-in-records/exceptions, view-dependent no-arg ops) MUST stay sibling stores / handles / free functions. Keeps §4 rung 1's incentive intact.
7. Of this thread's §5 open questions: Q5 is answered (the inverse exists by construction in the flat model); Q4 softens (a keyed method with a spec entry now has a correctness story, so SupportsUrlFor being method-shaped is less damning); Q1 splits per finding 5. And a new question 0 for the maintainer: what does wrap_kvs compile to at the endgame — is-a (the wrap_kvs will wrap the instance but self of instance is not wrapped #18 doc's Phase 3) or the flat engine (this doc's P2)? They currently contradict; a plausible synthesis is codec semantics → flat engine, class-decorator usage → is-a.

The doc's §9 rescopes migration honestly (P2 is not "wrap_kvs becomes a facade" — inner-layer wants_self/postget observe their layer's domain, which fusion cannot materialize; route 2 (mk_relative_path_store → Files) is separate), and §10 sets the mixed old/new stack policy (legacy Store = opaque leaf, with a warning).

[thorwhalen]

Decisions on the Option G open questions (maintainer, 2026-08-10)

All six questions from dol_issue86_design.md §11 are decided; §11 is now a decision record, and the code-level ones landed in #89:

• Q0 — wrap_kvs endgame: split synthesis. Codec/instance wrapping compiles to the flat engine; @wrap_kvs class-decoration becomes is-a — each mechanism owns the population it is uniquely correct for (backend-direct Key-transform wrappers delegate capability methods with the unmapped key (url_for etc. silently address the wrong object) #83 bodies vs outer-view wrap_kvs will wrap the instance but self of instance is not wrapped #18 bodies). This supersedes the "flat vs is-a" rivalry between this doc's P2 and the wrap_kvs will wrap the instance but self of instance is not wrapped #18 doc's Phase 3; the crisp fire-when rule is P2/P3 design work.
• Q1 — surface: private + facade. The engine stays private; MappingInterface (built-in spec) and kv_interface_wrap (wrap_kvs-shaped kwargs facade) shipped so the simple gesture never regresses. Export reconsidered after P1 adapter experience.
• Q2 — loudness default: 'exclude'. Wrap succeeds; using an undeclared attribute raises with guidance. 'raise' remains as strict mode. (Wrap-time raise on real leaves — dict: 11 publics, boto3: 122 — was judged to drive passthrough-as-habit.)
• Q3 — typed codecs: yes, now. Codec carries optional decoded_type/encoded_type; adjacent stack seams are validated at wrap time (outer encoded_type ≡ inner decoded_type), loud on mismatch, untagged unchecked. Tagging dol.trans.Codec waits for P2 (dependents gate).
• Q4 — eq/hash: outer-view eq, no hash when the spec'd surface covers __getitem__+__iter__ (a wrap equals a dict of its outer items; __hash__ is None). Store's own eq/hash incoherence queued for 0.4.
• Q5 — __class__ transparency: off; opt-in later, co-designed with Better way to control the wrapper class #5.

Next milestones per the (rescoped) plan: P1 — census-family adapters trial interface_wrap for spec-expressible capability surfaces (sibling stores stay blessed for refused shapes, per the §7 decision rule), plus the adapter-side conformance helper; P2 — the split-synthesis compatibility appendix (.store surface, signature graft, wrapped_self backref) before any wrap_kvs wiring.