Report: dol, from delegation traps to a typed boundary-codec architecture (the #86/Option G cycle, complete walkthrough) #90
thorwhalen
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
dol, from delegation traps to a typed boundary-codec architecture — the complete walkthrough
Part I — What dol is, and what it aims for
I.1 The promise
dol ("data object layer") is a pure-Python, zero-dependency
toolkit whose promise is: wrap any storage backend — files, S3, databases, zips, APIs,
plain dicts — behind the built-in Mapping interface, then layer transformations on top and
keep using the result as if nothing changed. The founding statement of intent is
discussion #35, "Present and Future of dol",
which names three principles:
user_store[user_id] = user, with the store responsible forserializing/encrypting/dispatching; the caller thinks in domain objects.
s[k],k in s,list(s),del s[k]) instead of a per-backend method zoo.Filesfor S3 without touchingbusiness logic.
I.2 The main features (before this work)
Collection → KvReader → KvPersister → Storeindol/base.py.Storeholds the four transform hooks(
_id_of_key/_key_of_id/_data_of_obj/_obj_of_data, defaulting to identity) and routesthe Mapping dunders through them.
wrap_kvs— the heart of the package(
trans.py#L1899):attach key/value transforms (
key_of_id,id_of_key,obj_of_data,data_of_obj, pluskey-aware
postget/preset) to any class or instance.store_decorator(
trans.py#L138)makes every wrapper usable four ways: class decorator, class factory, instance wrapper,
parameter-only factory.
Codec/ValueCodec/KeyCodecdataclasses with+composition(
trans.py#L3608)and the ready-made
ValueCodecs/KeyCodecscollections.Files/TextFiles/JsonFiles(
filesys.py),cache_this/cache_vals/store_cached(
caching.py),KeyTemplate/mk_relative_path_store(
paths.py#L1102).wrap_kvs— the blast-radiusnumbers that gate any change to the core (see the
architecture map).
The adapter family includes s3dol,
mongodol, azuredol,
sshdol, and more.
I.3 The mechanism underneath: delegation (has-a), via two routes
wrap_kvs(SomeClass)does not return a subclass ofSomeClass. It returns a generatedStoresubclass holding an instance ofSomeClassinself.storeand forwarding to it.Two independent routes do the forwarding:
Store.__getattr__(
base.py#L742)— runtime fallback, used by instance-wraps and by
mk_relative_path_store-builtclasses (which is how
dol.Filesitself is built:filesys.py#L632).DelegatedAttributedescriptors(
base.py#L252),installed one-per-attribute at class-wrap time by
delegate_to.Both routes return methods bound to the inner (leaf) object. Keep that sentence in mind;
it is the root of everything in Part II.
Part II — The limitations, and the ideal we stated
II.1 The bug family: one root cause, three symptoms
The promise of I.1 holds for the Mapping dunders and fails for everything else. Because
methods are delegated leaf-bound, two symmetric failures arise — consolidated in the
#86 body and the
#83 design study:
url_for,replace,delete_many) receives the outer, unmapped keyself[k]and bypasses the transforms (selfis the leaf)__signature__, breaking subclass signaturesNothing raises. Capability detection can't see it (
@runtime_checkableProtocols checkpresence, not key-correctness). And because there are two delegation routes, every fix
historically had to be applied twice — "the second application is the one people forget"
(#83 study, §5.3).
The scale was measured, not assumed: a 13-package census (indexed from
#83) found the family overwhelmingly latent —
12 survey claims of live damage were refuted — but with real teeth where it does bite:
cosmodol.CosmosItems.replacesilently overwrites a different, real document under a keycodec;
pydrivedol.GDReader.get_urlreturns the wrong file's URL and grants permissions onit. The trigger for the whole cycle was s3dol's measured failure table in
s3dol discussion #14: under a
logs/scope,url_forsigns the wrong object,delete_many(['a.txt'])destroys a root object outsidethe scope, and
prefixes()leaks a neighboring tenant.II.2 The other cracks found while looking
@wrap_kvs-decorated classes,dol.Files, and stacked codec wraps all failpickle.dumps, with one root cause: the generated classes are anonymous orname-shadowed (the reduce payload references a class no longer reachable at its dotted
name — e.g.
filesys.py#L632-L641where the decorated
Filesshadows the original).__getitem__(12× dict at onelayer, 90× at six), and full iteration 34–42× dict at one layer — each instance-level wrap
adds two delegation objects, not one.
__or__/copy/fromkeysget copied as raw delegates —wrapped | otherreturns untransformed inner data silently.Store.__eq__compares outer (transformed) views whileStore.__hash__hashes the inner store.
wrap_kvsalready shipsingoing_key_methods/outcoming_key_methods— the embryo of "declare which methods take keys" — but it is untested
(TODO at
trans.py#L2051)and verified broken for leaf-defined methods on both wrap routes (its generated body
calls
super(), which never reaches delegated methods).ADR-0006):
dol's shipped
prefixed/suffixedkey codecs silently corrupt out-of-band keys(decode
'z.txt'→''), and alogs/scope can surface a neighbor tenant's keys asreadable and writable.
II.3 The prior option space (A–F) and the blessed answer of the day
The #86 body (and
#83 study §3)
had already priced six options:
wrapped_selfselfinside method bodies via a weakref registry (base.py#L350)_key_methods = {'url_for': 0}url_for(store, k)walking the chainStorelayers; "more reliable, not reliable"self= the wrappersuper()crashes; "do not re-propose without new information"BucketUrls[k]— route keyed capabilities through__getitem__wrap_kvs(Class)returns a real subclassII.4 The stated ideal — the carry-forward list
#83 study §5
wrote down what any "great redesign" must deliver. Abbreviated:
Store.wrapbreaks the signature of subclasses. #6 are three symptoms of it);None;Storelayers exist in the wild — refuse rather than guess;(azuredol ≈ 0 and is clean.)
And the wish that seeded this cycle had been on the books since 2023: the maintainer's
reading-list comment in #35
lists wrapt's "Proxies and Wrappers"
as material to study "before making any dol re-design choices". Related aspirations sat in
#24 (backend fast-path hooks, e.g. a
_filter_method that
filt_iterwould consult), #49(context managers / idempotent connections), and
#46 (dunder-forwarding hooks; stdlib
codecs).II.5 The proposal that started this cycle
The maintainer proposed (2026-08-10; now on record in the
#86 Option G comment):
with codecs hooked up automatically per a specification — e.g. a Protocol class
annotated with
KT/VT;for speed and validation;
Part III — The work
III.1 Method
The cycle ran as: understand (six parallel deep-reads: wrapt semantics,
base.pydelegation,
wrap_kvsinternals, the prior design docs, s3dol's real needs, and empiricalprobes of pickle/isinstance/performance) → design draft → adversarial refute panel
(four independent reviewers briefed to break the draft, with runnable counterexamples) →
prototype → independent code review (a separate adversary on the diff) → decide
(six explicit maintainer decisions). Both adversarial rounds drew blood; the corrections are
flagged inline in the
design doc
(its §12 is the verification log).
III.2 The evidence that shaped the design
On wrapt (verified by running wrapt 2.3.0):
ObjectProxygives isinstance/__class__transparency, attribute passthrough, ~80hand-implemented dunders — but inside a wrapped object's method,
selfis the wrappedobject. The proxy intercepts only the first hop. wrapt has exactly dol's wrap_kvs will wrap the instance but self of instance is not wrapped #18 hole;
a proxy on its own fixes nothing about inside→out.
NotImplementedError: object proxy must define __reduce__()); a subclass__reduce__round-trips fine — a posture worth copying andimproving on.
__iter__regret: one eagerlydefined dunder permanently poisoned
hasattrduck-typing, fixable only via a new base classten years later. Lesson: a proxy must mirror the leaf's capability surface, never
resurrect a deliberately-missing method (dol's
_filt_itercurrently does resurrect__len__— same bug class).wrapt's C extension for attribute forwarding (~83 vs ~445 ns) — a dependency-free carrier
costs nothing.
On typing introspection (verified on 3.10 and 3.12): stdlib
get_type_hints+inspect.signature+get_origin/get_argsrecover, per method, exactly where roleTypeVars occur — bare,
Iterable[KT],tuple[KT, VT],Optional[KT],dict[KT, VT],nested
Iterator[tuple[KT, VT]]. No blockers; several version-specific traps (3.10'simplicit-
Optionalfor= Nonedefaults;Callable[[KT], …]hiding roles in a list arg)were found by the adversarial rounds and are handled.
On s3dol's real needs (from
ADR-0011
and a survey of s3dol/base.py):
the shape vocabulary a real mechanism must express — scalar key args, iterables of keys,
key-keyed dicts, keys in returned iterables (outward mapping — which didn't exist in dol
at all), keys embedded in records and exceptions, key-space-shifting returns (
sub(),handles), cross-keyspace values. Crucially, ADR-0011 had rejected declarative registries
for their silence-by-omission failure mode — so loudness had to be the mechanism, not an
afterthought.
III.3 Option G — the design
A wrap is
(leaf, spec, stack)— one proxy object, however many layers. Implemented indol/_interface_wrap.py(private, experimental, stdlib-only), tested in
dol/tests/test_interface_wrap.py(45 tests). The pieces:
InterfaceSpec):a Protocol annotated with role TypeVars (
KT,VT, or any TypeVar — roles are general),compiled per method into "role at path" plans. Mapping dunders are ordinary spec
entries — no privileged surface (carry-forward item 2 satisfied). A dict form
(
{'__getitem__': {0: 'KT', 'return': 'VT'}}) serves annotation-free use. Roles are matchedby TypeVar name, not identity — deliberate, because dol itself ships two distinct
KTobjects and identity-matching would silently classify a user's same-named
KTas"not a key" (found by the panel).
{role: Codec(encoder, decoder)}layers. Re-wrapping builds a newproxy with a copied-and-extended stack over the same leaf — never a wrapper-of-wrapper.
Compilation fuses encoder/decoder pipelines per role
(
_compile_method_plan).Consequences:
inner_most_keybecomes a total fold; the inverse mapping exists byconstruction (carry-forward 4); the leaf is a strong structural reference,
w.__wrapped__(carry-forward 6); stacked wraps pickle by construction (value-based__reduce__, no dynamic class in the payload); and per-op cost is one boundary hop at anydepth — measured 3.6× faster than nested
wrap_kvsat depth 6 (675 vs 2426 ns getitem;206 vs 700 µs iterating 1000 keys).
InterfaceProxy):a generated-and-cached class per
(leaf type, spec, surface)containing exactly thespec'd methods the leaf actually has — wrapt's lessons (capability mirroring,
_self_state naming, loud pickling) without its transparency goal. A dunder outside the spec
simply doesn't exist on the proxy:
proxy | otherraisesTypeErrorinstead of silentlyreturning raw inner data.
attributes (policy-controlled), unannotated parameters inside spec'd methods refuse at
compile time (
UnderAnnotatedSpecError— a spec author who writesurl_for(self, k)forgot the annotation and
kwould silently get outer keys), and unsupported shapes refuse(
UnsupportedSpecShape) — including roles insideCallable[[KT], …](contravariantpositions would hand inner keys to outer callbacks) and
**kwargs: VT.The two headline semantic results:
at the proxy boundary and
selfinside bodies stays the leaf, so internalself.x()calls never re-cross the boundary. Verified with a counting encoder: exactly one
invocation through a spec'd method that internally calls another spec'd method.
correct under the stack — iff the codec laws hold (decoder total + injective on the
leaf's actual keys; encoder/decoder mutual inverses; container args copied at the
boundary). The unconditional version was refuted by the panel using dol's own
suffixedcodec on an out-of-band key. The laws are §2.5 of thedesign doc.
And the headline architectural finding (new evidence that corrects the prior corpus):
is-a wrapping does not fix #83 for backend-direct method bodies. A body like
cosmodol.replacethat hands its key straight to the backend still receives the outer keyunder is-a — hooks on the MRO don't help a body that never routes through them (verified by
panel probe; an is-a hook additionally shadows a leaf that owns its own
_id_of_key, theexact shape of s3dol's
S3BucketReader).So F and G serve disjoint populations — F fixes outer-view bodies (#18), G fixes
leaf-domain bodies (#83) — and #86's option-F verdict "both #83 and #18 disappear" is
retired. This is why Option G is not an interim stand-in for is-a.
III.4 What the adversarial rounds broke (and what was done about it)
The four-lens panel
(§12 rows 6–12) and the independent code review (row 14) produced, among ~30 verdicts:
w.__wrapped__is not the Enabling (optional)KeyPathability to write when path doesn't already exist #16/Makingconditional_data_transbetter, and generally; recursively applying wrappers #10 write-back boundary — thepath_set_writebackengineneeds the raw-scalar surface (leaf plus the value-codec slice of the stack), verified
by running the real engine both ways. The flat design needs a stack-slicing accessor
(future work; the draft's "two O(1) accessors" claim was corrected).
filt_iterchanges the key set and
cached_keysis stateful; neither is a codec, so mixedcompositions still nest. The layer vocabulary needs a filter-layer kind (P3).
wants_selftransformtoday receives the layer-i wrapper, and a layer-i
postgetobserves intermediate keys —observable API that a fused pipeline never materializes. P2 is rescoped to
flat-equivalent configurations with a loud fallback; route 2
(
mk_relative_path_store→Files) migrates separately or not at all.__reduce__fixes thestacked-anonymous-class failure by construction, but
Filesand decorator-form wrapsfail because the leaf's own class is name-shadowed — a dol-side fix, not a wrapper-side
one.
*keys: KTvar-positional was silently corrupted (tuple encoded as one key); a 3.10-only failure
(builtin slots like
dict.__getitem__have no text signature there); dict-form specscrashed on unpickle/copy; explicit dunder access (
s.__contains__('outer_key')) escaped tothe leaf and answered in the wrong key domain;
iter()on a__getitem__-only proxyinvented integer-key iteration via the legacy sequence protocol; keyword calls of the fast
path worked only by parameter-name coincidence;
Iterableargs were downgraded to one-shotmaps.
proxy, rebinding
selfto the proxy makes internal self-calls re-cross the boundary →double-encode (probe-verified) — replacing option D's innermost-binding defect with a
violation of G's own invariant. The rejection stands; only its premise inventory changed.
III.5 What shipped
(earlier in the same investigation): export + harden
inner_most_key; fixcontent_urland
filesyskey validation (v0.3.59).Option G design doc
(next section).
#86 Option G,
#86 decisions,
#24,
#35,
#49,
#46.
III.6 The six decisions — options, trade-offs, and what we chose
All decided by the maintainer 2026-08-10; recorded in
design doc §11
and the #86 decisions comment.
Decision 0 — What does
wrap_kvscompile to at the endgame?The biggest fork, forced by the disjoint-populations finding (III.3).
wrapped_self(with its verified silent hole) becomes the permanent answerChosen: split synthesis — codec/instance wrapping compiles to the flat engine;
@wrap_kvsclass-decoration becomes is-a. Rationale: the two failure populations areprovably disjoint, so a single winner was never available; the split assigns each mechanism
its provable domain.
Decision 1 — Public surface now?
Chosen: private + facade. Shipped in PR #89:
MappingInterface(the built-in Mapping-shaped spec — deliberately only the six routed dunders; the mixin
methods each need their own vocabulary decision) and
kv_interface_wrap(
wrap_kvs-shaped kwargs —id_of_key/key_of_id/data_of_obj/obj_of_data— compiledonto the engine), so the one-line gesture is preserved.
Decision 2 — Loudness default for undeclared leaf attributes
The panel's strongest UX finding: wrap-time refusal (
'raise') fires on 11 undeclaredpublics for a plain dict and 122 for a boto3 client, so users would habitually reach for
'passthrough'— after which omission is silent again ("loud one keystroke deep").'exclude'default ✅'raise''raise'+ spec generatorChosen:
'exclude', with'raise'retained as strict mode.Decision 3 — Typed codecs now?
The codec laws (III.3) are only enforceable if codecs carry type information — which two
in-tree TODOs had already asked for
(
trans.py#L3606,kv_codecs.py#L310).Chosen: now. Shipped: optional
decoded_type/encoded_typeon the engine'sCodec,with
_validate_stack_seamsrefusing loudly when adjacent layers disagree at their seam; untagged stays unchecked.
Tagging
dol.trans.Codecwaits for P2 (32-dependent gate).Decision 4 — eq/hash policy
No member of {
__eq__,__hash__,__len__-under-filtering} is derivable from rolemapping, and today's
Storeis incoherent (outer-view eq, inner-store hash).wrap == {'a': 1}works); coherent; mutable-mapping conventionChosen: outer-view eq (when the spec covers
__getitem__+__iter__),__hash__ = None.Store's own incoherence queued for 0.4.
Decision 5 —
__class__transparency (isinstance-as-leaf)wrapt proves it's feasible; it also changes capability detection and collides with
#5 (wrapper-class control).
Chosen: off today; co-design with #5.
Part IV — Conclusion: what is still ahead
IV.1 The staged plan, as it now stands
P1 — adapter trials (next). The census-family adapters (
cosmodol.replace,pydrivedol.get_url,sshdol.sync_to's siblings, s3dol'surl_for) trialinterface_wrapon their spec-expressible capability surfaces. The governing rule, adopted in the design
doc's §7: spec expressibility is the line — shapes the spec can express may be methods
iff spec'd; shapes it refuses must stay sibling stores, handles, or free functions (this
preserves the "fewest keyed methods" incentive of
#86 §4 and of
ADR-0011).
P1 also owes: the adapter-side reflective conformance helper (the two-desks answer to
"the guard is worth more than the registry"), and a decision on the cheap mitigation of the
P1 enforcement hole — during P0–P1 nothing protects users of the legacy wrap path, so
teaching
wrap_kvsto warn when key-wrapping a store whose class declares a spec is worthconsidering.
P2 — the split-synthesis build-out. Before any actual
wrap_kvswiring, acompatibility appendix must answer, in writing: does the facade's product subclass
Store? What does.storereturn when there is no layer-below object (≥51 lines in dol coreread
.store)? Is the #6 signature graftpreserved? Does the engine register the
wrapped_selfbackref(today's blessed #18 fix silently degrades to identity on engine-built wraps otherwise —
probe-verified)? How are the two
__missing__paths (leaf-native with inner keys;Store-level with outer keys) reproduced? And the fire-when rule for the split itself:
exactly which invocation forms of
wrap_kvsroute to the flat engine vs is-a. The hardtechnical constraint to respect: fusion is only semantics-preserving when no inner layer
observes its own domain — inner-layer
wants_self/postget/presetneed a loud fallbackto nesting.
P3 — the larger vocabulary and the is-a build-out.
role → codec: at minimum a filter-layerkind (for
filt_iter's key-set change — which would also finally give#24's
_filter_fast-path hook a home);decide
cached_keysseparately.@wrap_kvsdecorator population, folding in thenewly found hook-collision hazard (a leaf owning
_id_of_keygets shadowed by a naiveis-a hook) and the constraints already listed in the
#18 staged plan
(MRO on stacked wraps, dropping
Store.__getattr__on is-a, ABC registration, thedependents gate before any default flip).
write-back boundary is the leaf plus the value-codec slice of the stack — a derived
proxy over the same leaf with a stack prefix. Needed before the path/view layers
(
add_path_access,_paths_core.path_set_writeback)can sit on flat wraps.
__class__transparency, co-designed with #5.IV.2 Known-unsolved issues (the honest list)
keys: Iterable(author forgot[KT]) is indistinguishable from a deliberately role-free iterable. The unannotated caseis loud (
UnderAnnotatedSpecError); the under-parameterized case remains a review-timejudgement — exactly where ADR-0011 D5
put it.
key-space-shifting returns (
sub(), handle-returning methods, #10's territory),cross-keyspace values, keys embedded in returned records and exception payloads
(laziness compounds the latter: the raise site has no boundary frame), untyped operation
batches, view-dependent no-arg operations (
sync_to), replace-the-write strategies,pushdown.
Filesand decorator-form wraps fail because theleaf class itself is name-shadowed
(
filesys.py#L632-L641) —fixing name-shadowing in dol is its own work item, independent of the wrapper.
Storelayers around/under a flat proxy degrade theflat-model guarantees; today the engine warns and treats a legacy Store as an opaque leaf.
The eventual absorb-or-refuse policy belongs in P2's appendix.
distinct
KTobjects), so two genuinely different roles sharing a name is a userresponsibility.
Store's own eq/hash incoherence — queued for 0.4 alongside the decided outer-viewpolicy.
dol.trans.Codectype tags — decided in principle (Decision 3) but gated on P2's32-dependent test gate.
does) — accepted follow-up, documented in
design doc §12 row 14.
IV.3 The decision queue (what needs a human)
get(default-domain problem),keys()(
KeysView[KT]),update(**kw)(keyword-names-as-keys) needs its own call before thebuilt-in spec can grow.
interface_wrap/kv_interface_wrap(after P1).wrap_kvsconsult specs from P1 on?).sibling store" (= #10) so ADR-0011-style
siblings stop requiring parallel manual wrapping?
IV.4 One-paragraph summary
dol's core promise — wrap and keep using — held only for the Mapping dunders because
delegation hands every other method the wrong key domain in one direction and the wrong
selfin the other. This cycle turned the maintainer's wrapt/typed-spec/flat-compileproposal into Option G: a spec-carried, flat-compiled, boundary-codec engine that is
loud where the old machinery was silent, picklable where it wasn't, ~3.6× faster at depth,
supplies the long-missing inverse key mapping — and, per the headline finding, is not
made redundant by is-a wrapping, because the two fix provably disjoint populations. Six
decisions now steer the follow-through: split the endgame between the two mechanisms, keep
the engine private behind a simple facade, refuse at the moment of danger, type the codecs,
give Mapping-shaped wraps Mapping equality, and keep
__class__honest until #5 is designed.Appendix — link index
_interface_wrap.py—Codec+tags L151 ·InterfaceSpecL297 ·_validate_stack_seamsL457 ·InterfaceProxyL666 ·interface_wrapL813 ·MappingInterfaceL958 ·kv_interface_wrapL996 · testsStore.__getattr__L742 ·DelegatedAttributeL252 ·delegate_toL416 ·delegator_wrapL491 ·wrapped_selfL350 ·wrap_kvsL1899 · broken*_key_methodsL1917 ·store_decoratorL138 ·CodecL3608All reactions