v2.4.0
Big release. Four DX features (N+1 detector, GenericRelation
prefetch, only/defer with select_related, Pydantic Create/Update
schemas), full vector search support (pgvector + sqlite-vec)
under one dorm.contrib.pgvector module, and a chain of latent
bug fixes the new test passes surfaced.
Added — Vector search (dorm.contrib.pgvector)
Same module covers both backends:
| Backend | Column type | Distance functions |
|---|---|---|
| PostgreSQL | vector(N) |
<-> / <=> / <#> operators |
| SQLite | BLOB |
vec_distance_L2 / vec_distance_cosine |
VectorField(dimensions=N)— declares the column. Vendor-aware
db_type(vector(N)on PG,BLOBon SQLite) and
get_db_prep_value(text on PG, packed little-endian float32
on SQLite). Validates length on write; round-trips
list[float]/tuple/numpy.ndarray/ pgvector's
Vector/ SQLite BLOB / memoryview on read.L2Distance/CosineDistance/MaxInnerProduct— distance
expressions that compile per-vendor: pgvector operators on PG,
vec_distance_*calls on SQLite. Compose with
annotate()+order_by()for kNN.MaxInnerProduct
raisesNotImplementedErroron SQLite (sqlite-vec ships no
negated-IP function — useCosineDistanceover normalised
embeddings instead).HnswIndex/IvfflatIndex— index helpers for the two
pgvector ANN methods.opclass=defaults to
vector_l2_ops; method-specific storage parameters
(m=,ef_construction=,lists=) flow through to a
WITH (k = v, …)clause via the newIndex.with_options
hook. (Vector indexes on SQLite require sqlite-vec'svec0
virtual table — not yet wrapped.)VectorExtensionmigration operation:- PG →
CREATE EXTENSION IF NOT EXISTS "vector"forwards,
DROP EXTENSION IF EXISTS "vector"backwards. - SQLite → loads sqlite-vec into the migration's connection AND
flipsSQLiteDatabaseWrapper._vec_extension_enabledso every
future connection auto-loads the extension.
- PG →
load_sqlite_vec_extension(conn)— public helper for manual
loading from app boot / ASGI lifespan / worker startup.dorm makemigrations --enable-pgvector <app>— CLI flag that
writes the boilerplate migration callingVectorExtension().djanorm[pgvector]extra — installs both thepgvector
Python package (psycopg adapter) andsqlite-vec(loadable
extension binary) so a single install line covers both backends.- Step-by-step documentation at
docs/pgvector.md
with separate sections for the PostgreSQL and SQLite paths.
Added — N+1 detector
dorm.contrib.nplusone.NPlusOneDetector— context manager
that hookspre_query, normalises every executed SQL to a
parameter-stripped template, counts hits per template, and (in
strict mode) raisesNPlusOneErrorat exit when a template
fired more thanthresholdtimes.raise_on_detect=False
produces a non-fatal report viadetector.report()for
staging-style auditing.assert_no_nplusone()— pytest-friendly helper.
NPlusOneErrorsubclassesAssertionErrorso pytest's
traceback rewriting kicks in.- DDL noise (
CREATE/DROP/ALTER/PRAGMA,
transaction control) filtered by default; caller can override
viaignore=. - Identifier characters (
"authors"."id") preserved during
normalisation; only string / numeric / NULL literals
collapse to?so unrelated queries don't bucket together.
Added — prefetch_related on reverse GenericRelation
- Reverse polymorphic relations now batch.
Article.objects.prefetch_related("tags")resolves every tag
pointing at every article in a single SELECT
(content_type = ct AND object_id IN (…)), groups by
object_id, and stamps each article's manager cache. The
GenericRelationmanager's.all()reads from that cache
before falling back to the live query — same contract as
Django's prefetch + reverse-FK relation. - Async parity via
_aprefetch_generic_relation. Prefetch("tags", queryset=…)honoured — the user-supplied
filters / ordering / select_related are AND-ed onto the
content_typepredicate.
Added — only() / defer() compose with select_related
- Dotted paths now restrict the related projection.
Author.objects.select_related("publisher").only("name", "publisher__name")
emits a single LEFT OUTER JOIN that pulls only"authors"."id",
"authors"."name","publishers"."id", and
"publishers"."name"— previously the projection-restriction
short-circuited the JOIN entirely, silently degrading to a
per-row N+1 on the related side. - PK is always implicit so the hydrated related instance keeps a
valid identity even when only non-PK columns were listed. defer("publisher__bio")is the inverse: keeps every related
column except the named ones.- Bare names (
only("name")) keep their legacy semantics:
parent-only restriction. Mixed (only("name").defer("publisher__bio"))
works because the two sets live in separate state buckets.
Added — Pydantic Create / Update schema helpers
create_schema_for(model)— drops auto-incrementing PKs and
GeneratedFieldcolumns automatically (server-controlled),
keeps required fields required, propagates real defaults. Plain
BaseModelsubclass suitable as a FastAPI request body.
Default class namef"{Model.__name__}Create".update_schema_for(model)— every remaining column becomes
T | Nonewith defaultNone(PATCH semantics). Built via
pydantic.create_modeldirectly so the field-default
propagation can be neutralised — a column with
default=Falsemustn't advertise that default to the client
when partial-update semantics say "no change". Constraint
translation (max_length,ge=0,Literal[…]for
choices, …) still applies — PATCH bodies aren't a free pass.- Both helpers accept
name=,exclude=(extends the auto-PK
drop), andbase=(customBaseModelancestor).
Fixed — select_related + only() was silently a no-op
- The SQL builder used to short-circuit the
select_relatedJOIN
wheneveronly()/defer()was active, on the (incorrect)
assumption that the restricted projection would conflict with
the aliased SR columns. The aliased columns
("_sr_<path>_<col>") are namespaced by construction; emitting
both is safe. ORDER BYnow qualifies the parent column with the table name
whenever any JOIN is in flight (WHERE-derived or
select_related). Previously it qualified only whenself.joins
was truthy and missed the SR case, producingambiguous column name: idonce a parent"id"and a related"id"both
appeared unqualified.
Fixed — select_related + WHERE column shared with related table
- WHERE column now qualified when
select_relatedis active.
Author.objects.filter(name="x").select_related("publisher")
previously emittedWHERE "name" = …without a table prefix —
PostgreSQL raisedcolumn reference "name" is ambiguousand
SQLite silently picked the parent's column._resolve_column
now treats bothself.joinsandself.select_related_fields
as triggers for qualification.
Fixed — DecimalField returned float on SQLite
DecimalField.from_db_valuecoerces the cursor's value to
decimal.Decimal. SQLite stores NUMERIC with REAL affinity,
sosqlite3returned floats — the field's annotation
promisedDecimalbut the runtime value didn't match,
breaking arithmetic withTypeError: unsupported operand type(s) for +: 'float' and 'decimal.Decimal'. PG's psycopg
adapter already returnedDecimal; anisinstanceguard
keeps that path zero-cost.
Fixed — Migration autodetector missed non-type-changing field edits
- The autodetector compared only
field.db_type(connection),
which collapses different dimensions of a SQLite VectorField to
the sameBLOB(and similarly drops nullability / default /
max_lengthtweaks for fields whose SQL type stays the same).
The diff now also compares the writer's serialised output, so
any edit the migration writer would emit differently triggers
anAlterField.
Fixed — AlterField on SQLite was a no-op
- SQLite's
ALTER TABLEdoesn't supportALTER COLUMN, so
the operation silently did nothing.AlterField.database_forwards
now follows SQLite'srecommended rebuild recipe <https://www.sqlite.org/lang_altertable.html#otheralter>_:
create a new table with the up-to-date schema, copy the column
intersection, drop the old, rename, and recreate any
Meta.indexesdeclared on the model. PG path now also flips
nullability (DROP NOT NULL/SET NOT NULL) on top of the
existingALTER COLUMN TYPE.
Fixed — Migration writer dropped VectorField(dimensions=…)
- The writer's
_serialize_fieldhad no branch for
VectorField, so generated migrations emitted
VectorField()and crashed at import with
TypeError: __init__() missing 1 required positional argument: 'dimensions'. The branch + the matching_FIELD_IMPORTS
entry are now in place.
Tests
tests/test_pgvector.py— ~100 cases. Unit-level field /
expression / index helper coverage that runs on plain SQLite
without any extension; integration cases that auto-skip when
pgvector / sqlite-vec aren't loaded but verify round-trip + kNN
ordering + index DDL when they are.tests/test_bug_hunt_v2_4.py— 80 cases targeting historically
bug-prone ORM patterns (empty-input boundaries, NULL FK +
select_related, Q-object identities, Decimal precision
round-trip,get_or_create/update_or_createsemantics,
CASCADE depth, NULL ordering, unicode / SQL-special characters,
M2M idempotency, transaction rollback nesting, F-expression
equality, UNIQUE conflict →IntegrityError, FK column
ordering). Both the Decimal and ambiguous-column bugs above
were caught by tests in this file.
Added — N+1 detector
dorm.contrib.nplusone.NPlusOneDetector— context manager
that hooks the :data:pre_querysignal, normalises every executed
SQL to a parameter-stripped template, counts hits per template,
and (in strict mode) raises :class:NPlusOneErrorat exit when a
template fired more thanthresholdtimes. Strict mode is the
default;raise_on_detect=Falseproduces a non-fatal report
viadetector.report()for staging-style auditing.assert_no_nplusone()— pytest-friendly helper that wraps
the detector in strict mode and surfaces the violation as a
regularAssertionError(so pytest's traceback rewriting
works). Designed for use inside individual test functions.- DDL noise filtered by default —
CREATE/DROP/
ALTER/PRAGMA/ transaction control statements are
ignored so test-fixture bookkeeping doesn't trip the detector.
Caller can override viaignore=for custom suppression. - Identifiers (
"authors"."id") are preserved verbatim during
normalisation; only string / numeric / NULL literals get
collapsed to?. This keeps unrelated queries from accidentally
bucketing together while still catching parameter-only variations
of the same shape.
Added — prefetch_related on reverse GenericRelation
- Reverse polymorphic relations now batch.
Article.objects.prefetch_related("tags")resolves every tag
pointing at every article in a single SELECT
(content_type = ct AND object_id IN (…)), groups by
object_id, and stamps each article's manager cache. The
GenericRelationmanager's.all()reads from that cache
before falling back to the live query — same contract as Django's
prefetch + reverse-FK relation. - Async parity via
_aprefetch_generic_relation, dispatched
alongside the existing async prefetch coroutines through
:func:asyncio.gather. - Custom
Prefetch(queryset=…)is honoured — the user-supplied
queryset's filters / ordering / select_related survive; the
content_type+object_id__inpredicates are AND-ed onto it.
Added — only() / defer() compose with select_related
- Dotted paths now restrict the related projection.
Author.objects.select_related("publisher").only("name", "publisher__name")
emits a single LEFT OUTER JOIN that pulls only"authors"."id",
"authors"."name","publishers"."id", and
"publishers"."name"— previously the projection-restriction
short-circuited the JOIN entirely, silently degrading the query
to a per-row N+1 on the related side. - PK is always implicit. Even when the user lists only non-PK
related columns, the related model's PK column is added to the
projection so the hydrated instance keeps a valid identity. defer("publisher__bio")is the inverse: keeps every related
column except the named one(s).- Bare names retain legacy semantics.
only("name")(no
dotted path) restricts only the parent model; the related side
loads in full. The parent and related restrictions live in
separate state buckets so combining them
(only("name").defer("publisher__bio")) works.
Added — Pydantic Create / Update schema helpers
create_schema_for(model)— drops auto-incrementing PKs
andGeneratedFieldcolumns automatically (server-controlled),
keeps required fields required, propagates real defaults. Plain
BaseModelsubclass suitable as a FastAPI request body. Default
class namef"{Model.__name__}Create".update_schema_for(model)— every remaining column becomes
T | Nonewith defaultNone(PATCH semantics). Built via
:func:pydantic.create_modeldirectly so the field-default
propagation can be neutralised — a column withdefault=False
must not advertise that default to the client when partial-update
semantics say "no change". Constraint translation (max_length,
ge=0,Literal[…]forchoices, …) still applies — PATCH
bodies aren't a free pass on validation.- Both helpers accept
name=,exclude=(extends the auto-PK
drop), andbase=(customBaseModelancestor).
Fixed — select_related + only() was silently a no-op
- The SQL builder used to short-circuit the
select_relatedJOIN
wheneveronly()/defer()was active, on the (incorrect)
assumption that the restricted projection would conflict with the
aliased SR columns. The aliased columns ("_sr_<path>_<col>")
are namespaced by construction, so emitting both is safe. The
short-circuit is gone; SR JOINs now run alongside any projection
restriction. ORDER BYnow qualifies the parent column with the table name
whenever any JOIN is in flight (WHERE-derived or
select_related). The previous "qualify only whenself.joinsis
truthy" rule missed the SR case and producedambiguous column name: idonce a parent"id"and a related"id"both
showed up unqualified.