Graph-object binding: attribute access costs 11x-18x vs plain pydantic; descriptor binding is a faster strict superset with no syntax change

[simontaurus]

Summary

Reading any attribute of a LinkedBaseModel (including plain, non-reference fields) costs roughly 11x (pydantic v1) to 18x (pydantic v2) more than the equivalent plain pydantic model, because attribute access is intercepted unconditionally. Writes, linked-field access and query construction are slower still.

This is fixable without changing the declaration syntax. Installing a data descriptor per x-oold-range field keeps today's exact model syntax (standard annotations, plain List[...] for to-many), reaches parity with plain pydantic on ordinary fields, and is faster on every other operation as well.

Two declaration front-ends share one implementation, so both are supported:

• implicit - annotated field plus a range keyword (what the code generator emits today, unchanged),
• explicit - Link / LinkList descriptors for hand-written models.

Verified by script, the descriptor binding is a strict superset of the current implementation: it satisfies every requirement the shipped binding satisfies, plus three it does not (attribute projection on link lists, validated json_schema_extra, no pydantic monkeypatch).

Requirement matrix

Produced by examples/check_binding_features.py, which exercises each requirement rather than asserting it. Each variant runs in its own subprocess.

requirement	shipped v1	shipped v2	auto (implicit)	auto (explicit)	Ref[T]
syntax_unchanged	ok	ok	ok	FAIL	FAIL
build_by_iri	ok	ok	ok	ok	ok
build_by_object	ok	ok	ok	ok	ok
lazy	ok	ok	ok	ok	ok
real_object (isinstance holds)	ok	ok	ok	ok	FAIL
polymorphic (subclass via type IRI)	ok	ok	ok	ok	FAIL
batched (N links, one call)	ok	ok	ok	ok	FAIL
cached	ok	ok	ok	ok	ok
mutation	ok	ok	ok	ok	FAIL
link_validated	ok	ok	ok	ok	ok
list_lookup links["ex:t2"]	ok	ok	ok	ok	FAIL
list_filter links[T.label == "two"]	ok	ok	ok	ok	FAIL
list_projection links.label	FAIL	FAIL	ok	ok	FAIL
serialize_iri	ok	ok	ok	ok	ok
query_dsl Cls[Cls.f == v]	ok	ok	ok	ok	FAIL
typed_extras (validated)	FAIL	FAIL	ok	ok	FAIL
no_monkeypatch	FAIL	FAIL	ok	ok	ok

auto (implicit) and auto (explicit) are not competing options: they are the two
declaration front-ends of the same variant C, sharing one descriptor implementation and one
registry, and they can be mixed in a single class. syntax_unchanged is FAIL for the explicit
form by definition - it is deliberately a different declaration style, offered for
hand-written models - and FAIL for Ref[T] because the wrapper appears in the annotation.
Every other row is identical between the two front-ends, as expected.

Requirement legend

requirement	meaning
syntax_unchanged	standard annotations, no wrapper type in the declaration
build_by_iri	construct a link from an IRI string
build_by_object	construct a link from a model instance
lazy	no backend call before first access
real_object	access returns the real target, isinstance holds
polymorphic	resolves to the actual subclass via its type IRI
batched	an N-item list resolves in ONE backend call
cached	a second access does not re-resolve
mutation	assignment replaces the link and invalidates the cache
link_validated	the linked object is validated by its own model on construction
list_lookup	links["ex:t2"]
list_filter	links[T.label == "two"]
list_projection	links.label collects the attribute across items
serialize_iri	serialisation emits IRIs for links
query_dsl	Cls.field == v and Cls[cond]
typed_extras	validated json_schema_extra (OoldExtra)
no_monkeypatch	importing the module does not patch pydantic.fields.FieldInfo (checked in a subprocess)

Performance matrix

100,000 iterations per operation, best of 5, each variant in its own subprocess (importing oold.model monkeypatches pydantic.fields.FieldInfo process-wide and would otherwise contaminate the baselines). Times in ms; (x) is relative to plain pydantic v2 plain-read. Link reads are warm (already resolved).

variant	plain read	plain write	link read	link write	query build
plain pydantic v1	3.7 (0.7x)	46.9 (9.3x)	na	na	na
plain pydantic v2	5.1 (1.0x)	24.6 (4.9x)	na	na	na
gated __getattribute__	18.3 (3.6x)	69.3 (13.7x)	na	na	na
shipped v1	56.4 (11.2x)	972.5 (192.5x)	434.7 (86.0x)	1597.4 (316.1x)	283.1 (56.0x)
shipped v2	91.1 (18.0x)	561.8 (111.2x)	825.5 (163.4x)	1162.5 (230.1x)	484.0 (95.8x)
auto (implicit)	5.1 (1.0x)	50.1 (10.2x)	6.3 (1.3x)	270.8 (55.2x)	194.6 (39.7x)
auto (explicit)	5.1 (1.0x)	47.9 (9.8x)	6.4 (1.3x)	277.5 (56.6x)	194.7 (39.7x)
explicit Ref[T]	4.9 (1.0x)	24.4 (5.0x)	5.8 (1.2x)	27.6 (5.6x)	na

Versus shipped v2 the descriptor binding is 19x faster on plain reads, 11x on plain writes, 131x on link reads, 4.2x on link writes and 2.5x on query construction.

On writes: pydantic itself defines __setattr__, so writes are Python-level in every variant (plain v2 writes already cost 4.9x a read). The descriptor variant adds one frame plus a dict lookup, landing at 9.5x - about 2x plain pydantic, but 12x cheaper than today. Classes with no link fields need no __setattr__ override at all.

Ref[T] is fastest on link access only because it returns the wrapper without resolving; it is not semantically comparable (isinstance fails).

Two further details from a separate run isolating the interception cost on plain reads:

gating strategy	vs plain v2
best case (closure frozenset, no attribute lookup on the fast path)	3.13x
realistic (per-class set, needs a type(self) lookup)	4.53x

Note also that plain pydantic v1 attribute access is faster than v2 (0.7x), which is why each
binding is compared against its own pydantic baseline rather than a single one.

Environment: Python 3.11.6, pydantic 2.12.0, Windows.

Root cause

oold/model/__init__.py intercepts attribute access unconditionally:

class LinkedBaseModel(BaseModel, GenericLinkedBaseModel, metaclass=LinkedBaseModelMetaClass):
    def __getattribute__(self, name):          # runs for EVERY attribute
        ...
        if hasattr(self, "__iris__"):
            if name in self.__iris__ and len(self.__iris__[name]) > 0:
                ...                            # sync backend I/O inside the getter
        result = BaseModel.__getattribute__(self, name)

Plus a process-wide pydantic.fields.FieldInfo = OOFieldInfo monkeypatch at import; a metaclass that also overrides __getattribute__ (for the query DSL) and therefore needs a _constructing guard to avoid corrupting pydantic's metaclass bookkeeping; and a parallel __iris__ side-dict duplicating field state.

Could the interception simply be gated on range annotations? Partly. Early-exiting for non-link fields removes most of the work (18x down to ~3.6x), but not the call: defining __getattribute__ at all forces the interpreter to invoke a Python-level function on every access instead of using the C-level slot, which alone costs ~3.6x. The gate shrinks the body, not the call. The descriptor is the same gate implemented in C.

Variants

A. Current implementation

from typing import List, Optional
from pydantic import Field
from oold.model import LinkedBaseModel

class Person(LinkedBaseModel):
    id: str
    name: Optional[str] = None
    knows: Optional[List["Person"]] = Field(
        None, json_schema_extra={"range": "Person"}
    )

p = Person(id="ex:p1", knows=["ex:p2", "ex:p3"])   # by IRI, or by object
p.knows[0].name                                     # real object, lazily resolved
isinstance(p.knows[0], Person)                      # True
p.knows["ex:p2"]                                    # IRI lookup
Person[Person.name == "John"]                       # query DSL

Pros: no special syntax (plain annotations, List[Person] for to-many - exactly what datamodel-code-generator emits); real objects on access; batched and lazy resolution; query DSL; IRI lookup and filtering on link lists.

Cons: 11x-18x on every attribute read and up to 316x on link writes; global FieldInfo monkeypatch; json_schema_extra is a raw untyped dict, so {"rnge": ...} fails silently; __iris__ duplicates field state; resolution is hidden in a sync getter, so it cannot be awaited.

B. Gate the interception on range annotations

Smallest possible change, keeps syntax A exactly.

class LinkedBaseModel(BaseModel, ...):
    __link_names__ = frozenset()   # computed per class from range annotations

    def __getattribute__(self, name):
        if name in type(self).__link_names__:
            ...            # existing resolution logic, unchanged
        return object.__getattribute__(self, name)

Pros: no syntax change, small diff, 18x down to ~3.6x on plain reads.

Cons: cannot reach parity; leaves the monkeypatch, __iris__ and the untyped extras untouched.

C. Descriptor binding (recommended)

After pydantic finishes building the class, scan model_fields for a range annotation and install a data descriptor per link field. A data descriptor takes precedence over the instance __dict__, so link reads go to the descriptor while every other field keeps native pydantic access.

Both declaration front-ends are supported and can be mixed in one class:

from oold.experimental.auto_descriptor_binding import (
    AutoLinkedModel, Link, LinkList, OoldField,
)

class Person(AutoLinkedModel):
    id: str
    name: Optional[str] = None

    # IMPLICIT: unchanged syntax; equivalent to
    # Field(None, json_schema_extra=OoldExtra(range="Person"))
    knows: Optional[List["Person"]] = OoldField(default=None, range="Person")

    # EXPLICIT: for hand-written models. No doubled type argument needed -
    # the subscript carries the static type, __orig_class__ the runtime target.
    employer = Link(Organization)
    friends  = LinkList["Person"]()

Usage and query patterns are unchanged from A:

p = Person(id="ex:p1", name="Alice", knows=["ex:p2", "ex:p3"], employer="ex:acme")

p.link_iris("knows")               # ['ex:p2', 'ex:p3']  - no backend call
p.knows[0].name                    # 'Bob'  -> ONE batched call resolves the list
isinstance(p.knows[0], Person)     # True
p.knows["ex:p3"]                   # IRI lookup
p.knows[Person.name == "Bob"]      # filtering
p.knows.name                       # attribute projection -> ['Bob', 'Carol']
p.model_dump(exclude_none=True)    # links collapse back to IRIs

Person.name == "John"              # Condition(field='name', operator=eq, ...)
Person[Person.name == "John"]      # query by condition
Person["ex:p1"]                    # query by IRI
Employee.salary > 100              # inherited fields
Person.knows == "ex:p2"            # link fields, straight off the descriptor

Implementation sketch:

class AutoLinkedModel(BaseModel, metaclass=LinkedQueryMeta):
    _links: Dict[str, Any] = PrivateAttr(default_factory=dict)
    _link_cache: Dict[str, Any] = PrivateAttr(default_factory=dict)
    __link_fields__: ClassVar[dict] = {}

    @classmethod
    def __pydantic_init_subclass__(cls, **kwargs):
        super().__pydantic_init_subclass__(**kwargs)
        links = dict(getattr(cls, "__link_fields__", {}))
        # explicit form: descriptors declared in the class body
        for klass in reversed(cls.__mro__):
            for key, value in vars(klass).items():
                if isinstance(value, _AutoLink):
                    links[key] = value
        # implicit form: annotated fields carrying a range keyword
        for name, field in cls.model_fields.items():
            extra = field.json_schema_extra
            if isinstance(extra, dict) and (extra.get("x-oold-range") or extra.get("range")):
                target, many = _extract_target(field.annotation)  # Optional[List[X]] -> (X, True)
                descr = _AutoLink(name, target, many)
                setattr(cls, name, descr)      # data descriptor shadows the field
                links[name] = descr
        cls.__link_fields__ = links

    def __init__(self, **data):
        lf = type(self).__link_fields__        # route link kwargs before validation
        link_data = {k: data.pop(k) for k in list(data) if k in lf}
        super().__init__(**data)
        for k, v in link_data.items():
            lf[k].__set__(self, v)

    def __setattr__(self, name, value):
        # targeted: pydantic writes model fields straight into __dict__, which
        # would bypass a data descriptor's __set__ and leave the cache stale
        descr = type(self).__link_fields__.get(name)
        if descr is not None:
            descr.__set__(self, value)
        else:
            super().__setattr__(name, value)

The query DSL moves from __getattribute__ (every access) to __getattr__ (a fallback, only when lookup fails). Pydantic v2 removes field names from the class namespace, so Person.name fails naturally and lands there at no cost to anything else:

class LinkedQueryMeta(ModelMetaclass):
    def __getattr__(cls, name):
        # CRITICAL: never call getattr(cls, ...) here. cls.model_fields is a
        # property that itself calls getattr -> infinite recursion.
        if name.startswith("_"):
            raise AttributeError(name)
        for klass in cls.__mro__:
            fields = klass.__dict__.get("__pydantic_fields__")
            if fields and name in fields:
                return FieldProxy(name)
        raise AttributeError(name)

    def __getitem__(cls, item):
        return cls.oold_query(item)

For link fields no metaclass is involved: the descriptor's __get__(None, owner) returns the descriptor on class access, so comparison operators live directly on it.

Pros: no syntax change (implicit form); parity on plain reads and faster on every other operation; real objects, polymorphic dispatch, batching, caching, rich list operations, query DSL; validated extras; no monkeypatch; the _constructing guard disappears.

Cons: the descriptor shadows the pydantic field, so the parent's field validation is bypassed and link kwargs are routed in __init__ (the current implementation already special-cases them similarly). Link fields no longer live in __dict__. Metaclass __getitem__ still shadows generic subscripting (Model[int]) - unchanged from today.

On validation: the linked object is still validated at construction of the linked class, which is where its constraints live - a dict-valued link is constructed through the target model, so {"label": "no id"} raises for a required id. What is lost is only the parent field's own annotation check.

D. Explicit Ref[T] (opt-in handle)

class Person(LinkedModel):
    knows: Optional[List[Ref["Person"]]] = None

p.knows[0].iri              # 'ex:p2'   without resolving
p.knows[0].resolve().name   # explicit
await p.knows[0].aresolve() # async

Pros: resolution is visible, batchable and awaitable - none of which A can express.

Cons: syntax and semantic change: p.knows[0] is a Ref, not a Person, so isinstance fails and list operations do not apply. Suitable as an opt-in handle where explicit or async resolution is wanted, not as the default.

E. Rejected: Annotated wrapper that reads as the target type

knows: Optional[List[Linked["Person"]]] = None   # Linked[X] == Annotated[X, ...]

Type checkers report Person, but the runtime value is a Ref, so isinstance(p.knows[0], Person) is False. A static type not backed by the runtime value; do not use.

Typed json_schema_extra

The raw dict can be replaced by a validated class, but it must subclass dict: pydantic merges extras via isinstance(json_schema_extra, dict), so a plain BaseModel is accepted at declaration and then silently dropped from the schema.

class OoldExtraModel(BaseModel):
    """Constraints live here - real pydantic validation."""
    model_config = ConfigDict(populate_by_name=True, extra="allow")
    range: str = Field(alias="x-oold-range", min_length=1)
    required_iri: Optional[bool] = Field(None, alias="x-oold-required-iri")


class OoldExtra(Dict[str, Any]):
    def __init__(self, *, range: str, required_iri: Optional[bool] = None, **vendor: Any):
        data = {"x-oold-range": range}
        if required_iri is not None:
            data["x-oold-required-iri"] = required_iri
        data.update(vendor)                            # x-jedison-*, x-osl-*, ...
        model = OoldExtraModel.model_validate(data)    # validate via dict, not kwargs,
        object.__setattr__(self, "_model", model)      # so aliases stay out of the signature
        super().__init__(model.model_dump(by_alias=True, exclude_none=True))

    @property
    def range(self) -> str: return self._model.range
    @property
    def required_iri(self) -> Optional[bool]: return self._model.required_iri

OoldExtra(range="")          # ValidationError: String should have at least 1 character
OoldExtra(range=123)         # ValidationError: Input should be a valid string
Person.model_json_schema()   # ... 'x-oold-range': 'Person'  -> preserved
type(Person.model_fields["knows"].json_schema_extra)   # OoldExtra
extra.range                  # typed read (str), instead of extra["x-oold-range"]

Type checking (pyright), confirmed: e.range is str, e.required_iri is bool | None, OoldExtra(range=123) and OoldExtra() are errors.

Caveats: validation happens in __init__ rather than by pydantic validating the field itself; pass the payload to model_validate as a dict rather than as aliased kwargs, otherwise type checkers reject range= as "No parameter named"; extras must stay JSON-serialisable for schema export.

Static typing

Confirmed on pyright and mypy for the explicit form; the implicit form is plain annotations and so types natively.

p.knows          -> List[Person]        (LinkList["Person"]() - subscript only)
p.knows[0].name  -> str
p.employer       -> Organization | None (Link(Organization)   - argument only)
p.knows[0].nope  -> error: Cannot access attribute "nope" for class "Person"

LinkList["Person"]() needs no second argument: the subscript carries the static type, and the runtime target is recovered from __orig_class__.

CPython and Rust optimisation potential

Already applied - warm link reads at native speed. The descriptor is deliberately a
non-data descriptor (it defines __get__ but not __set__) and stores the resolved value in
the instance __dict__. Because an instance dict entry shadows a non-data descriptor, every
subsequent read is a plain C-level dict lookup that never re-enters Python - the
functools.cached_property pattern. Writes remain intercepted by __setattr__, which pops the
cached entry to invalidate it.

The effect is large: caching in a pydantic PrivateAttr instead costs a Python-level
__getattr__ call per read, which is what made link reads slow.

link read (warm)	time	vs plain field
data descriptor + PrivateAttr cache	336.0ms	31.8x
non-data descriptor + __dict__ cache	10.5ms	1.00x
plain pydantic field (baseline)	10.6ms	1.00x

That is a 32x improvement on the hot path, and it takes link reads from 33.6x to 1.3x in the
full matrix above.

Remaining CPython headroom (not yet applied):

• Query construction, ~6.6x available. Condition is a pydantic BaseModel, so every
  Cls.field == value pays full model validation: 200.1ms vs 30.4ms for an equivalent
  __slots__ class (200k iterations). This would take query build from 39.7x to roughly 7x. It
  touches the public oold.backend.interface API, so it is a deliberate change rather than a
  free win.
• Link writes, currently 55.2x. Dominated by Ref construction and PrivateAttr access on
  the write path; __slots__ on Ref and avoiding the private-attr lookup should recover much
  of it.
• Plain writes, 10.2x vs 4.9x for plain pydantic. Entirely the extra __setattr__ frame.
  Classes with no link fields need no override at all, so the base class should install it
  conditionally.

Rust potential. After the fix above, the binding hot path is already C-level (an instance
dict lookup), so there is essentially nothing left for Rust to win there, and pydantic's
validation core is Rust (pydantic-core) already. The real Rust opportunities are elsewhere in
the stack:

• JSON-LD processing. pyld is pure Python and dominates RDF export: to_jsonld() costs
  120.7 us/op versus 19.7 us/op for to_json(), i.e. 6x, essentially all of it context
  expansion. A Rust-backed JSON-LD processor would attack the single most expensive operation in
  the library.
• RDF and SPARQL. rdflib is likewise pure Python; pyoxigraph (Rust, oxigraph) is a
  drop-in-ish alternative for graph storage and SPARQL in the RDF backends.
• Schema processing / code generation (bundling, $ref resolution over large schema graphs)
  is another candidate, though it is build-time rather than runtime.

Priority: the JSON-LD/RDF layer is where Rust would pay off, not the object binding.

Design rationale

The shipped design makes every attribute transparently resolve. Python has no cheap
whole-object proxy, so that choice forces __getattribute__ plus a metaclass. Per-field
descriptors give the same transparency for just the link fields, at native cost for everything
else - which is why they reach parity while gating cannot.

Cross-language, every ecosystem that handles this well either makes resolution explicit
(Rust/TreeLDR IdRef<T>, Java OGM sessions, Datomic pull) or has a language-level proxy
that makes transparency cheap (JavaScript Proxy). Python has neither at whole-object level,
but the descriptor protocol provides exactly the per-field equivalent, and Ref[T] covers the
explicit camp for async and batched control. Supporting both front-ends therefore matches the
two durable designs found elsewhere rather than picking one.

Remaining work

• pydantic v1: the prototype is v2-only (__pydantic_init_subclass__). model/v1/__init__.py is a full parallel implementation and the package generator emits both v1 and v2, so a v1 path or a decision to drop v1 is required.
• Public-API equivalence with the shipped LinkedBaseModel (to_json / to_jsonld / from_json / from_jsonld / cast / BaseController / Model["iri"]) must be demonstrated before adoption so osw-python is unaffected.
• The prototype has no dedicated unit-test suite yet; it is currently covered by the two matrix scripts.

Proposal

Adopt the descriptor binding with both front-ends: the implicit, annotation-based form as the default (unchanged syntax, so generated packages are untouched), and the explicit Link / LinkList form for hand-written models. Keep Ref[T] as an opt-in handle for explicit or async resolution.

Reproduce

python examples/check_binding_features.py     # requirement matrix
python examples/bench_binding_variants.py     # performance matrix

Correction: an earlier revision of this issue stated that the current implementation issues N backend calls for an N-item list. That was wrong - the shipped binding already batches list resolution into one call. Batching is parity, not a gain.