Design: layered transformations and the unmapped-key problem

[thorwhalen]

Splitting this out of the v1 redesign (#11) because it is a dol-wide problem, not an s3dol one, and because the v1 design only sidesteps it rather than solving it.

The problem

dol's model is "wrap a base store with layers of transformations". Key transforms are applied by the wrapper's __getitem__ / __setitem__ / __delitem__ / __contains__ / __iter__. Every other method is invisible to the transformation layer — dol.base.Store.__getattr__ returns the bound leaf method, so it receives the outer, unmapped key.

For a backend adapter this is severe, because the interesting capabilities are exactly the non-dunder methods.

Measured (dol 0.3.58), store scoped to logs/, in a bucket that also holds logs2/leak and a root a.txt:

w['a.txt']              -> b'IN'                   correct
url_for('a.txt')        -> https://s3/a.txt        WRONG (want /logs/a.txt) — signed, silent
sub('x/')               -> store over bucket ROOT   scope escape
handle('a.txt')         -> handle on root a.txt     wrong object
info('a.txt')           -> KeyError                  for a key that IS present
prefixes()              -> ['logs/', 'logs2/']       leaks the sibling it exists to hide
delete_many(['a.txt'])  -> ROOT a.txt DESTROYED      silent destruction of the wrong object
len(w)                  -> 1                         a __len__ we deliberately omitted

Minimal repro:

from dol import KeyCodecs

class WithUrl(dict):
    def url_for(self, k): return f"https://x/{k}"

w = KeyCodecs.prefixed('a/')(WithUrl)({'a/b': 1})
w['b']          # -> 1              correct, prefix applied
w.url_for('b')  # -> 'https://x/b'  WRONG: should be 'https://x/a/b'

Nothing raises, and isinstance(w, SupportsUrlFor) stays True — @runtime_checkable checks method presence only.

There is a second, independent failure mode: pushdown is impossible. Store.__iter__ calls self.store.__iter__(), so there is no channel by which a key-wrapper hands its prefix to the leaf's ListObjectsV2(Prefix=…). Every prefix-scoped listing becomes a full-bucket scan — measured at 22 LIST requests where a leaf-owned prefix costs 1.

Filed upstream as i2mint/dol#83 (delegation) and i2mint/dol#82 (the related prefix-corruption bug).

What v1 does, and why it isn't a solution

v1 sidesteps it: the prefix lives in the leaf, so there is no key-mapping seam between a capability method and the wire (ADR-0001 §Why the prefix lives in the leaf). That fixes the prefix case only. A user who wraps an s3dol store with any dol key codec still gets a silently wrong url_for. Every *dol adapter with a keyed capability method has the same latent bug.

Options

A — status quo + documentation

Prefix in the leaf; document the trap; use inner_most_key(wrapped_self(self), k) inside any capability method that might be wrapped.

Cheap. Leaves the footgun loaded for users and sibling packages. Note the natural form is itself a trap: inner_most_key(self, k) returns None (inside a delegated method self is the unwrapped leaf), so the URL becomes https://…/None with no exception.

B — declarative key-method registration

The leaf declares which methods take a key and where; dol's wrapper machinery generates mapped delegates.

class BucketReader(...):
    _key_methods = {'url_for': 0, 'info': 0, 'handle': 0, 'sub': 0, 'delete_many': ('iter', 0)}

General; fixes the family at once. But a method the author forgets to declare fails the same silent way. Mitigable with a reflective test that enumerates public methods and fails on any undeclared one — which is the kind of guard that actually holds.

C — free functions instead of methods

s3dol.url_for(store, k) rather than store.url_for(k). The function resolves the key through the whole wrapper chain once, then calls the leaf.

This is already dol's own idiom — dol.content_url(store, ref_or_key), get_content, put_content, add_content all take the store as first argument, and dol/content.py explicitly frames it as solving the problem beside the interface rather than inside it.

Composes at any wrapper depth; no delegation at all, so nothing to get wrong. Costs ergonomics, and dol.SupportsUrlFor currently expects a method — though both can coexist (method on the leaf for the unwrapped case, free function as the safe general form).

D — bind delegated methods to the outer store

Change Store.__getattr__ to return a delegate bound to the wrapper, so self._id_of_key(k) inside a leaf method resolves through the full chain.

Addresses the root cause rather than each symptom. But it is a behavioural change in dol's core with a wide blast radius (~42k downloads/month, many dependents), and it only works if leaf methods consistently call self._id_of_key. Closely related to dol#18.

E — capabilities as parallel Mappings

Turn each keyed capability into a Mapping view sharing the key space — store.urls[k], store.info[k], store.handles[k] — so key transformation happens through the Mapping protocol the wrapper already handles correctly. Requires the wrapper to re-wrap Mapping-valued attributes with the same key codec (declarative, à la B: _key_mapped_attrs = ('urls', 'info', 'handles')).

Most dol-native of the options: it turns delegation into composition. It also collapses three of the v1.x deferrals in #12 (presigned-URL store, ObjectInfo-from-LIST, handles) into one mechanism. Costs: more objects, and it doesn't cover non-keyed methods (delete_many, prefixes).

Provisional lean

C as the general safe form, E for capabilities that are naturally keyed, with B as the dol-level mechanism that makes E's re-wrapping declarative. D is the most principled but the riskiest to land in dol.

The pushdown half needs its own answer regardless — probably a hint protocol (__iter__(self, *, prefix_hint=…)) or an explicit iter_prefix on the leaf that the wrapper knows to route to. Worth deciding whether that belongs in this design or separately.

Questions I'd want settled

1. Is the ergonomic loss of C acceptable, given dol already does it for content?
2. Is E worth the extra objects, given it also buys three deferred features?
3. Should D be attempted in dol at all, or is the blast radius disqualifying?
4. Does pushdown get solved here, or is a full-bucket scan an accepted cost for wrapped stores?

[thorwhalen]

Resolved — ADR-0011, merged in #17

Decision: Layer B carries no key-taking public methods. A keyed capability becomes a
sibling store over the same key space — s3dol.handles(store)[k], .urls, .info — keyed
through __getitem__, which is the one thing dol maps correctly at every wrapper depth.
Non-keyed operations (sub, prefixes, delete_many, delete_bucket) are free functions.
url_for survives as a single guarded method purely to satisfy dol.SupportsUrlFor.

That is close to the provisional lean above (C + E, with B) — refined by evidence: E's value
comes from being a store, not an attribute; C shrinks to the non-keyed operations; B
contributes only its guard.

What changed the answer

Every option in this discussion was tested rather than argued, and three of the four premises
I started from turned out to be wrong.

Option D was already rejected upstream, in dol/misc/docs/dol_issue18_design.md — 26 sites
surveyed — and for a better reason than blast radius: rebinding binds self to the
innermost Wrap, so under a Pipe stack it does not fix the case it exists to fix, and
stacked-codec writes gain a partial-transform corruption surface. Q3 answers itself.

The option list was missing the terminal fix. dol#18's recommended direction is is-a
wrapping, which resolves dol#18 and #6 together. So the criterion became correct on dol 0.3.x
and harmlessly redundant under is-a — not permanent.

Option E's stated mechanism is false. A wrapper does not re-wrap a Mapping-valued
attribute: store.urls returns an inner-keyed mapping under both class- and instance-wrap. As
an attribute E needs the same key resolution as everything else. As a sibling store it is
correct by construction. Those are different designs and I had conflated them.

Option C's cited prior art has the bug. dol.content_url resolves with a flat
getattr(store, 'url_for')(key) — no chain walk — so through a key wrap it returns a URL for
the unmapped key. The idiom is prior art; the resolution is not.

And the escape ADR-0006 prescribed is itself unsafe. inner_most_key(wrapped_self(self), k)
is silently wrong when nothing holds a reference to the wrapper — the delegated bound method
holds none, and the weakref cleanup then removes the evidence. Because the prefix lives in the
leaf, the wrong answer is a plausible str, so a type check cannot catch it. Reproduced on
CPython 3.10–3.14.

I then drafted "free functions everywhere", and an adversarial review refuted that too: with a
hand-rolled non-Store layer in the chain, free functions return a plausible wrong key in three
configurations — two of which the method form gets right. All dol-shipped wrappers are safe;
dol/base.py ships a Delegator recipe of exactly the breaking shape.

Sibling stores need no key-resolution primitive at all, which is why they are the answer:
immune to both holes, verified unreferenced, under Pipe, under cached_keys, under a value
codec, and under the non-Store passthrough.

The reframe worth keeping

azuredol is robust because its container store has ~zero key-taking methods — the rich
surface lives on BlobHandle, keyed at construction — not because its prefix lives in the
leaf. Prefix-in-leaf is necessary, not sufficient. ADR-0001 had drawn the wrong lesson from
the same source.

Corrections to the framing above

• The measured table in the opening post is real, but s3_store(...).handle(k) is not an
  instance of the bug: s3_store returns a bare leaf or a value-codec wrap, both correct. Only
  a user-applied key codec triggers it.
• There are two delegation routes, not one: Store.__getattr__ (dol/base.py:742) for
  instance-wraps and mk_relative_path_store subclasses, and DelegatedAttribute.__get__
  (dol/base.py:279) for class-wraps. A fix covering one is a silent no-op on the other.
• The family census is overwhelmingly latent, not shipping-destructive — 13 packages
  re-verified, and 12 survey claims were refuted outright. Per-repo issues follow.

Answers to the four questions

1. Is C's ergonomic loss acceptable? Mostly moot — capabilities keep [k] indexing as
   sibling stores. C applies only to non-keyed operations.
2. Is E worth the extra objects? Yes, as stores. It is the only form needing no resolution
   primitive, and it collapses three ADR-0009 capability deferrals into one mechanism.
3. Should D be attempted in dol? No — rejected upstream with running-code evidence.
4. Does pushdown get solved here? Closed, not deferred (§D8): a prefix_hint protocol would
   have one implementer, violating ADR-0009's own rule. A user-stacked filter accepts a full
   scan; s3dol.sub(store, prefix) is the cheap path.

Follow-ups

Blocking upstream (i2mint/dol#84 open): export dol.dig.inner_most_key ✅, and fix
dol.content_url. Non-blocking: the wrapped_self lost-reference hole, and
dol.filesys.is_valid_key, which is confirmed-live broken. Per-package issues from the census
are being filed now, indexed from i2mint/dol#83.

Next: #15, credential and endpoint resolution.