Skip to content

v2.5.0

Choose a tag to compare

@rroblf01 rroblf01 released this 02 May 08:57
· 69 commits to main since this release

Minor release. Two opt-in features land alongside the v2.4.1
bug-hunt corpus, plus a follow-up audit pass that closed five
issues (B1, B4, B5, B7, B9) found while writing coverage tests.

Fixed — bulk write ops now invalidate the cache

  • QuerySet.update / delete / bulk_create /
    bulk_update (and the async counterparts) bypass
    post_save / post_delete per row — the previous
    cache layer left every cached queryset on the model
    populated for the full TTL after a bulk write. Now each of
    these methods schedules an invalidation through the same
    on-commit hook the per-instance signal handlers use.
  • No-op bulk calls (empty list to bulk_create /
    bulk_update, zero-row update / delete) DO NOT churn the
    cache or bump the version counter.
  • New helpers dorm.cache.invalidation.invalidate_model /
    ainvalidate_model for callers that route writes outside
    the standard QuerySet methods.

Hardened — opt-in strict signing key for multi-worker

  • Without CACHE_SIGNING_KEY / SECRET_KEY the cache
    layer falls back to a per-process random key. In
    multi-worker deployments this silently collapses the cache
    to per-worker visibility (other workers can't verify the
    signatures). New setting CACHE_REQUIRE_SIGNING_KEY (default
    False) refuses the fallback and raises
    ImproperlyConfigured on first cache use — recommended
    for any production-shaped deployment.

Fixed — cache key invariant under filter() kwarg ordering

  • filter(a=1, b=2) and filter(b=2, a=1) produced
    different SQL → different cache keys → halved hit rate for
    semantically identical queries. The cache-key digest now
    hashes a CANONICAL representation of the queryset state
    (sorted leaf-condition tuples, positional order_by,
    limit / offset / select_related / annotations / etc.) so
    iteration order doesn't perturb the digest. SQL emission is
    NOT sorted — a future query-plan tweak based on predicate
    order would break otherwise.

Fixed — libsql async wrapper detects event-loop change

  • LibSQLAsyncDatabaseWrapper stamped the loop on first
    open and reused the bound connection across every later
    call. Pytest-asyncio per-test loops + multi-loop ASGI
    workers triggered native crashes when a coroutine on a
    fresh loop reached into the prior loop's connection. The
    wrapper now checks the running loop on every _get_conn
    call and resets cached state (async conn, sync conn,
    executor, lock) when the loop changed — next acquire opens
    fresh.

Security — cache payloads now HMAC-signed

  • pickle.loads over Redis bytes is RCE if the cache is
    reachable by an attacker (multi-tenant Redis, no-auth
    deployment). Every cached payload now ships with an
    HMAC-SHA256 signature header (b"dormsig1:<hex64>:<pickle>").
    The signing key reads from settings.CACHE_SIGNING_KEY
    settings.SECRET_KEY → a per-process random key (with a
    one-time warning so the operator knows the cache isn't
    shared across workers).
  • Loads verify the signature with hmac.compare_digest BEFORE
    pickle.loads runs. Unsigned / tampered / truncated blobs
    are dropped silently — the queryset falls through to the
    database.
  • New settings: CACHE_SIGNING_KEY (recommended),
    CACHE_INSECURE_PICKLE (default False; opt-out for
    unsigned legacy caches you can't migrate).
  • Helpers exposed: dorm.cache.sign_payload /
    dorm.cache.verify_payload for callers building custom
    cache backends or test harnesses.

Fixed — Stale-read race between read and write

  • The naïve "read → DB fetch → store" flow could cache a stale
    row if a concurrent Model.save() invalidated the key
    between the reader's fetch and store steps. Closed with a
    per-model in-memory version counter: every save / delete
    bumps it; _cache_key includes ":vN:"; _cache_store_*
    re-reads the version after the DB fetch and stores under the
    (possibly bumped) key. A racing writer's bump now points
    later readers at the new key — the stale entry never gets
    written.
  • New: dorm.cache.model_cache_version /
    dorm.cache.bump_model_cache_version. Counter is
    process-local; cross-process coherence still goes through
    delete_pattern.

Fixed — parse_database_url("libsql:////abs") returned a relative path

  • The libsql URL parser stripped one slash too many for the
    four-slash absolute form. libsql:////var/data/db.sqlite
    produced var/data/db.sqlite — the open() landed next to
    the working directory instead of the intended /var/....
    Mirrors the sqlite branch's correct logic now: keep one
    leading slash for the absolute form, strip the lone slash
    for the relative one.

Fixed — ValuesListQuerySet._clone / CombinedQuerySet._clone dropped cache state

  • The base QuerySet._clone propagates _cache_alias /
    _cache_timeout, but the two subclass overrides forgot.
    qs.cache().values_list("name").filter(active=True) lost
    caching on the second clone — silent miss every call.

Fixed — execute_script async fallback corrupted quoted ; literals

  • The libsql async wrapper's executescript-not-available
    fallback split on bare ; — any DDL / DML containing a
    quoted ; (INSERT INTO t VALUES ('a;b'), identifier
    "weird;name") got partitioned mid-literal and the
    resulting statements failed at parse time. Replaced with a
    shared quote-aware helper (_split_statements in
    dorm/db/backends/sqlite.py) that ignores ; inside
    single- or double-quoted runs.

libsql backend — earlier round of fixes

  • async wrapper crashed on Python 3.14 + libsql_experimental
    due to native code issues and thread-safety violations
    (asyncio.to_thread fans across multiple workers; libsql
    connections aren't thread-safe). Migrated to pyturso
    (the official Turso Python SDK), pinned the async path to a
    single-thread ThreadPoolExecutor for remote / embedded-
    replica modes, and use turso.aio natively for local-only
    mode.
  • LibSQLDatabaseWrapper.sync_replica raised
    ValueError: Sync is not supported in databases opened in Memory mode when called against a local-only wrapper —
    pyturso exposes conn.sync even for memory DBs but
    rejects the call. Now skipped when SYNC_URL isn't
    configured.
  • Bind parameters: pyturso (and libsql_experimental) reject
    list, accept only tuple / Mapping. Both sync and
    async wrappers coerce in their execute-shaped methods.

libsql backend — features

  • libsql backend — talk to local files, remote
    Turso / sqld endpoints, or run as an embedded replica that
    syncs from a remote master. The dialect is SQLite-compatible,
    so the migration tooling and the SQLite branch of every
    compiler keep working untouched. Native vector support
    (F32_BLOB(N) + vector_distance_l2 /
    vector_distance_cos) is wired into VectorField so
    embeddings round-trip without the sqlite-vec extension.

Both features are gated behind optional dependencies — install
djanorm[libsql] and / or djanorm[redis] only when you
need them. djanorm itself imports without either client.

Added — libsql backend (dorm.db.backends.libsql)

  • ENGINE = "libsql" routes to LibSQLDatabaseWrapper
    (sync) and LibSQLAsyncDatabaseWrapper (async). Three
    modes share a single configuration shape:
    • Local file — drop-in SQLite replacement.
    • Self-hosted sqld (VPS) — typical production layout.
      SYNC_URL (https://...) + AUTH_TOKEN connect to
      your own server.
    • Embedded replica — local file + SYNC_URL keeps the
      file in sync with the remote master. sync_replica()
      on the wrapper triggers an explicit pull.
    • Turso Cloud — same wire protocol as self-hosted sqld;
      point SYNC_URL at libsql://<db>-<org>.turso.io.
  • Powered by pyturso — the official Turso Python SDK.
    Local-only async uses turso.aio natively; embedded
    replica / remote-only async runs the sync client on a
    dedicated single-thread ThreadPoolExecutor (pyturso
    connections aren't thread-safe, so the default
    asyncio.to_thread pool would fan calls across multiple
    workers and produce native crashes).
  • URL parser (parse_database_url) recognises libsql://,
    libsql+wss://, libsql+ws://, libsql+http:// and
    libsql+https://. Auth tokens come from the authToken
    query parameter; the optional NAME query parameter sets
    the embedded-replica file path.
  • Client lookup raises ImproperlyConfigured pointing at
    pip install 'djanorm[libsql]' if pyturso isn't
    installed.

Added — vector support on libsql

  • VectorField.db_type returns F32_BLOB(N) for
    vendor == "libsql". The packed-float32 wire format
    (already used by sqlite-vec) is reused — libsql's
    vector32(?) SQL function reads it directly.
  • L2Distance / CosineDistance compile to
    vector_distance_l2 / vector_distance_cos against
    vector32(?). MaxInnerProduct raises
    NotImplementedError (libsql ships no negated-IP function;
    use CosineDistance over normalised embeddings instead).
  • No VectorExtension() migration is needed on libsql —
    vector functions are built into the server.

Added — Result cache (dorm.cache)

  • QuerySet.cache(timeout=…, using="default") opts a
    queryset into result caching. Returns a clone (chaining is
    immutable, matching every other QuerySet API). Honoured by
    the sync iterator (for x in qs) and the async terminal
    (await qs).
  • Cache key is a SHA-1 of (final SQL, bound params)
    namespaced by f"dormqs:{app_label}.{ModelName}" so
    signal-driven invalidation can wipe every cached queryset for
    a model with a single delete_pattern call.
  • BaseCache defines the contract every backend implements:
    get / set / delete / delete_pattern (sync) and
    the matching a* async variants. Both return raw bytes so
    the queryset layer can pickle / unpickle on its own.
  • RedisCache (dorm.cache.redis) wraps redis-py for sync
    and redis.asyncio for async. Both pools are spun up
    lazily; LOCATION accepts every URL form redis-py knows.
    Every operation is wrapped in try / except — cache
    outages fall through to the DB silently, never propagate.
  • Auto-invalidation hooks (dorm.cache.invalidation)
    connect post_save / post_delete (sync + async) on
    first qs.cache() call, so projects that never opt into
    caching pay zero dispatch cost.
  • configure(CACHES={...}) invalidates the memoised cache
    instances so a mid-process settings swap doesn't keep the
    old client alive.

Added — settings

  • settings.CACHES (default {}). Same shape as Django:
    {alias: {"BACKEND": "dotted.path.Backend", "LOCATION": …, "OPTIONS": {…}, "TTL": 300}}.
  • settings.SEARCH_CONFIG (default "english") — already
    added in v2.4.1's R24 fix; mentioned here for completeness.

Optional dependencies

  • djanorm[libsql]libsql-experimental. Local file,
    embedded replica, or remote (Turso / sqld).
  • djanorm[redis]redis (sync + asyncio in one package).
  • djanorm[pgvector]pgvector only. The PostgreSQL
    psycopg adapter for VectorField. No longer pulls
    sqlite-vec
    — split into its own extra so a SQLite-only
    install doesn't drag in the PG adapter and vice-versa.
  • djanorm[sqlite-vec]sqlite-vec only. The SQLite
    loadable extension binary used by VectorExtension and
    VectorField on SQLite.
  • djanorm[vector] (new convenience meta-extra) → pulls both
    [pgvector] and [sqlite-vec] for projects targeting
    mixed PG/SQLite deployments.
  • djanorm[all] now pulls every extra alongside the existing
    set.

Tests

  • tests/test_libsql_v2_5.py — URL parsing, engine routing,
    local-file round-trip, vendor branch in VectorField /
    distance expressions, fallback error path when the client
    isn't installed.
  • tests/test_redis_cache_v2_5.pyqs.cache() clone
    semantics, hit / miss / store, key namespacing, sync + async
    round-trip, signal-driven invalidation, cache-outage
    fallthrough, helpful error when redis-py is missing.

Documentation

  • docs/libsql.en.md / docs/libsql.es.md — full guide
    covering local / remote / embedded-replica modes, async
    usage, vector support, migrations and limitations.
  • docs/cache_redis.en.md / docs/cache_redis.es.md
    configuration, qs.cache(...) semantics, async path,
    auto-invalidation contract, custom backend protocol, when
    caching helps and when it hurts.