Conversation
Stand up the measurement infrastructure that gates the rest of the browser-views performance work: - Add django-silk 5.5 to the dev group; install + route it to a separate ``silky`` SQLite DB via a new ``SilkRouter`` so profiler traces never touch the live DB. - Wire SilkyMiddleware below ServeStaticMiddleware so it only wraps the API stack, not static-file responses. - Expose /silk/ under the existing DEBUG URL block. - Add ``tests/perf/run_baseline.py``: hits the live dev DB via ``django.test.Client``, runs three cold/warm flow pairs (root browse, filtered search, series metadata), and writes a JSON baseline artifact. Cachalot + page-cache are invalidated before each cold pass so the numbers reflect actual DB work. - Add ``make perf-baseline`` target. - Commit the per-view analysis docs and initial baseline capture under ``tasks/browser-views-perf/`` for reference during the follow-on cleanup stages. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
) Bundle of small, surgical wins observed on the slimlib baseline: - Cache `libraries_exist` with a 60s TTL in Django's default cache; invalidate on Library save/delete via lazy-imported signal handlers. Drops a redundant `Library.objects.filter(...).exists()` from every browser response. - Short-circuit the four `_save_browser_*` code paths in `codex/views/settings.py` when nothing actually changed. Avoids gratuitous `.save()` calls that flush cachalot on no-op PATCH writes. - Dedupe the `add_group_by(qs)` call in `codex/views/browser/browser.py`: group once in `_get_common_queryset` (no-op for Comic), drop the second call in `_get_group_queryset`. - Memoize `get_max_bookmark_updated_at_aggregate` per `(model, agg_func, default)` on the view instance — the three callers (group_mtime, order, bookmark) now share one Aggregate. - Move the `bmua_is_max` flag off the per-row `Value` annotation and read it from `self.context["view"]` in the browser serializer. - `@lru_cache(maxsize=256)` on `_preparse_search_query`: extract to a pure module-level helper keyed on `(text, path_allowed)`; returns frozen tokens. - `@lru_cache(maxsize=512)` on `get_field_query`: cache the Lark-parsed Q tree, `copy.deepcopy` on return because `_hoist_filters` mutates `child.negated` downstream. - Stash the `BaseDatabaseOperations(None)` singleton as `_DB_OPS` in `search/field/expression.py`; `prep_for_like_query` doesn't use the connection. - Pre-filter the field loop in `filters/field.py` to keys actually set in the request — saves ~20 no-op calls per browser. - Delete `search/field/optimize.py` (`like_qs_to_regex_q` and friends were already unused — grep confirms only self-references). Slimlib cold baseline (3 flows, stage1.json vs Stage 0 baseline.json): - flow_a_root_browse: 21→18 SQL, 189.6→182.3 ms - flow_b_filtered_search: 21→17 SQL, 187.4→178.0 ms - flow_c_series_metadata: 34→31 SQL, 251.1→226.9 ms Warm paths unchanged (0 SQL, ~2 ms). Full tests + ruff pass. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Stage 0 wired a second SQLite DB (silky) for django-silk profiling traces
and a DATABASE_ROUTERS entry that routes silk app models there. But
ensure_db_schema only invoked `call_command("migrate")` without a database
arg, which only migrates the default DB. The router then blocks silk
migrations from running on default — so the silky DB was never populated,
and the first request through SilkyMiddleware failed with
"no such table: silk_request".
Mirror the pattern from tests/perf/run_baseline.py: after the default
migrate, also run `migrate silk --database=silky` when the silky DB is
configured. Guarded on `"silky" in connections.databases` so production
(DEBUG=False, no silky DB) is a no-op.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## PR 2a — Eliminate triple COUNT on the paginate path Each browse request ran three COUNT queries per section (groups & books): 1. The outer grouped COUNT in `_get_common_queryset`. 2. Paginator's internal COUNT triggered by `paginator.page()`. 3. An explicit `.count()` on the paginated slice. The outer COUNT is needed (sizing the paginator). The other two are redundant — the page row count is bounded by `per_page` and derivable from `end_index - start_index + 1`. - Shadow `Paginator.count` (a `@cached_property`) with the pre-computed total to skip Paginator's internal COUNT. - Derive page row count arithmetically from `Page.start_index()` / `end_index()`. - Pass `book_count` through `paginate()` alongside `group_count`; drop `book_qs.count()` on the opds2 path. - `_paginate_section` returns `(qs, count)` directly. Short-circuits on `total_count == 0` (avoids Paginator instantiation on empty sections) and preserves the EmptyPage warning branch. ## PR 2b — Short-TTL page mtime cache `BrowserView._get_page_mtime()` calls `get_group_mtime(page_mtime=True)` on every browse request. The query is a filtered Max aggregate that cachalot caches — but any write to Comic / Bookmark invalidates it, so bookmark-heavy usage forces recomputation. Cold-path silk traces show this aggregate at ~26ms on flow_a — the second-slowest query in the request. Add a 5s TTL cache layer gated on page_mtime=True. Key includes user, model, group, pks, page, and a hash of filter-affecting params (filters, search, q, order_by, order_reverse). The polling MtimeView path (no page_mtime) is unaffected, so frontend change-detection stays live. ## Measurements tests/perf/run_baseline.py on the slimlib DB. Cold = full cache invalidation; warm = cachalot populated. | flow | stage1 cold | stage2 cold | |-------------------------|------------------------|------------------------| | flow_a root browse | 18 queries / 182.3 ms | 16 queries / 135.6 ms | | flow_b filtered search | 17 queries / 178.0 ms | 15 queries / 130.3 ms | | flow_c series metadata | 31 queries / 226.9 ms | 31 queries / 229.9 ms | flow_a / flow_b: -2 queries, ~26% cold wall-time reduction. flow_c unaffected (metadata doesn't traverse paginate). PR 2b's benefit is a dogpile guard after cachalot invalidation — doesn't show in the harness (cold = both caches empty; warm = cachalot wins first). Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Stage 3 — cover fan-out: per-pk endpoint + pre-resolved cover_pk
Before stage 3, every cover card triggered a full BrowserAnnotateOrderView
pipeline to pick the representative comic pk. A default page does ~100 of
these in parallel. This PR collapses that into one correlated Subquery on
the browse response plus a thin per-pk cover endpoint.
Components
- Option A: annotate_order_aggregates(for_cover=True) drops the JsonGroupArray
+ page_count aggregates from the cover path — unused for picking a pk.
- Option B.1: BrowserView group cards get cover_pk / cover_custom_pk via
correlated Subquery that replicates CoverView.get_group_filter exactly
(direct fk match when dynamic_covers/Volume/Folder; sort_name fuzzy match
otherwise, correlated on _GROUP_BY columns so the same comic set as `ids`
is picked — no JSON parsing, no peer aggregate).
- New endpoints /api/v3/c/<pk>/cover.webp and /api/v3/cc/<pk>/cover.webp
serve already-resolved pks with a cheap single-row ACL probe and the
existing CoverPathMixin / CoverCreateThread pipeline.
- Frontend getCoverSrc prefers the new per-pk URL when cover_pk /
cover_custom_pk is on the card; falls back to the old group+pks URL
otherwise, so OPDS and search-active browses keep working.
- FTS skip: cover_pk annotation is skipped when params["search"] is set.
MATCH inside a correlated subquery re-scans the FTS5 index per outer
row (~900ms on a 100-group page). The old URL path still applies the
search filter per cover — same behavior, parallelized over HTTP.
Perf (slimlib dev DB, Flow D = browse + every card's cover):
Flow Before cold After cold Delta
A root browse 156.9ms / 15 q 180.5ms / 15 q +24ms / +0 q
B search 148.3ms / 16 q 132.4ms / 16 q -16ms / +0 q
C metadata 161.9ms / 31 q 165.0ms / 31 q +3ms / +0 q
D browse+covers 2311ms / 1216q 1161ms / 815 q -50% / -33%
Flow A takes a mild regression for the correlated cover subquery, but Flow
D — the realistic user wall-clock — drops by half and 400 SQL queries.
Subsequent cover fetches drop from ~90ms each (full pipeline) to ~5ms
(disk read + 1 ACL probe).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Stage 3 follow-ups: custom_cover URL, FTS pre-materialization, OPDS thin covers
Address review feedback on the cover fan-out collapse:
- Rename /api/v3/cc/<pk>/cover.webp → /api/v3/custom_cover/<pk>/cover.webp.
Descriptive over terse; matches the view name.
- FTS pre-materialization replaces the FTS-skip fallback. A correlated
MATCH re-scans FTS5 per outer row (~900ms on a 100-group page); the
old path worked around this by skipping cover_pk annotation on search
and sending every cover through the legacy pipeline once. We now
pre-select the FTS match set as a non-correlated sub-SELECT in the
outer cover subquery — SQLite materializes it once and each correlated
cover row lookup becomes an indexed pk filter. Cover_pk is annotated
on search responses, the thin endpoint handles each cover.
- OPDS now emits thin-endpoint cover URLs (v1 and v2):
- New OPDSComicCoverByPkView / OPDSCustomCoverByPkView wrap the browser
thin views with OPDSAuthMixin's Basic Auth.
- New opds:bin:cover_by_pk and opds:bin:custom_cover_by_pk URL names.
- v1 _cover_link picks: cover_custom_pk → custom thin; group=='c' → own
pk thin; cover_pk → thin; else legacy group+pks fallback.
- v2 _thumb always uses the thin endpoint (publications are Comic rows
so obj.pk IS the cover pk).
- OPDS inherits annotate_cover() via BrowserView._get_group_and_books,
so group rows already carry cover_pk / cover_custom_pk.
- tests/perf/run_baseline.py gains a flow_e (search + covers) to measure
the search path end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…eline (#579) - Fix folder cover_pk picking comics not actually in a folder's subtree by using the recursive ``folders`` M2M (same relation the browse filter uses) for the per-card cover subquery instead of the direct ``parent_folder`` FK. - Delete the legacy thick cover endpoint. Rename cover_by_pk.py to cover.py and drop the "_by_pk" suffix from view class names (CoverView, CustomCoverView, OPDSCoverView, OPDSCustomCoverView). Update URL configs, OPDS wrappers, and the frontend client. - Move all cover generation off the HTTP path. Add CoverCreateTask and enqueue it from the importer after bulk_create so new comics get thumbnails pre-warmed offline. When a cached thumb is missing the per-pk endpoint enqueues the task and responds 202 Accepted with Retry-After plus the missing-cover placeholder instead of synthesizing the WebP inline under a worker thread. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Browsers don't honor Retry-After on <img> elements and happily cache the placeholder served with the 202 response, so even after the cover thread finishes writing the real thumb the img src keeps rendering the stale placeholder bytes. - Backend sends Cache-Control: no-store on the 202 placeholder so the response isn't cached at that URL. - BookCover now probes the cover URL with fetch() on mount and, if it sees 202, waits Retry-After and bumps a reactive `retry` counter that becomes a cache-busting query param on coverSrc. v-img re-fetches with the new URL and gets the real cover once the cover thread is done. Retries are capped at 5 and aborted on unmount via AbortController. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Under a full cold-cache page load, 32 HTTP workers hitting the cover
endpoint at the same time as the cover thread was writing thumbs
returned 500s for a handful of covers. The endpoint used a three-step
exists()/stat()/read_bytes() sequence that could race against an
in-flight write, and save_cover_to_cache wrote directly to the target
path so readers could observe a truncated file mid-write.
- save_cover_to_cache now stages to a ``{name}.{pid}.tmp`` sibling and
renames with Path.replace, so a read either sees the pre-existing
state or the fully written file — never a partial one. Stale temps
are unlinked on failure.
- _get_cover_response collapses to a single read_bytes() that catches
FileNotFoundError and logs OSError, then falls through to the 202
path. No race window between stat and read.
- LIBRARIAN_QUEUE.put and the outer get() methods are wrapped in
try/except that log and return the placeholder, so a surprise
exception is never user-visible as a 500 and always leaves a log line.
- cleanup_orphan_covers runs a dedicated ``*.tmp`` sweep over both cover
roots before the orphan scan, so stale temps from crashed writers
don't accumulate.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
- Broaden alpha regex to match 1.11.0aN-amd64 / 1.11.0aN-arm64 so a release-cleanup run removes the whole alpha trio per version. - Add --orphans mode that walks every tagged manifest list via the ghcr.io OCI API, collects child digests, and deletes untagged versions not referenced by any of them. Cleans up old manifests left behind by tag overwrites and dangling buildx attestation manifests after their parent is gone. - --orphans-only skips the alpha pass for orphan-only sweeps. - Includes a sanity guard that aborts the orphan pass if the live digest set is empty, so a transient API glitch can't trigger a mass deletion. Default behavior is unchanged. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Granian + asgiref's SyncToAsync adapter shields the executor coroutine running a sync Django view. When a client disconnects mid-request, the outer task is cancelled; the inner future continues (sync work in a thread can't be cancelled externally), runs to completion, and is then unretrieved. asyncio's default exception handler logs that as "CancelledError exception in shielded future" at ERROR level — alarming in user-facing logs but functionally a no-op: the response was generated and quietly discarded because there's no client to send it to. Install a custom loop exception handler in _serve that drops CancelledError contexts and delegates everything else to the default. Surgical: only CancelledError is suppressed, real exceptions still surface. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…um spec (#702) * opds2: conform to Readium spec for belongsTo.series.position and publisher/imprint Closes #700. * belongsTo.series: emit `position` (was `number`, silently dropped by the serializer) so clients can read the issue number per series.schema.json. * publisher / imprint: render as Readium Contributor objects with browse links instead of bare strings, matching contributor.schema.json and the shape clients like Stump expect (mirrors readino.com OPDS2 output). * belongsTo.story_arc: fix the dict key (was `storyArc`, dropped by the serializer) so story arcs actually appear in manifests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * opds2: emit belongsTo.volume with name + position + browse link Volume is a top-level Readium belongs_to property (https://readium.org/webpub-manifest/schema/volume.schema.json) and codex already routes Volume as group "v" — but OPDS2 manifests never exposed it. Add ``_publication_belongs_to_volume`` mirroring the series helper: ``position`` carries the integer volume number, ``Volume.to_str`` formats the human-readable name (e.g. "v1" or "(2024)"), and the browse link points at the volume's comic list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * tasks: add handoff doc for Claude-worktree lint silent-skip issue Surfaced while finishing #702. Linters that walk up for ignore files (prettier, remark, eslint) hit the main checkout's ``.claude`` ignore and treat the worktree as ignored — remark errors loudly, prettier silently processes zero files. Documents the structural cause and the recommended fix (move worktrees out of ``.claude/``). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per beville's follow-up on #700: identifier ``url`` values are off-site representations of the same publication and belong in publication.links as ``rel=alternate`` (e.g. {"rel": "alternate", "type": "text/html", "href": "https://comicvine.gamespot.com/...", "title": "View on ComicVine"}). The base view runs a per-comic query for the manifest path; the feed view overrides with a batched UNION query keyed by comic pk so a full feed page stays at one identifier query regardless of book count (matching the existing credits / subjects batching). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…UG (#704) A truncated zlib stream in the file cache (left behind by a write killed mid-flush) was bubbling up as a 500 with a scary `zlib.error: Error -5` traceback into user-facing logs. ResilientFileBasedCache now treats a corrupt entry as a miss, deletes the bad file, and logs at DEBUG. Also silence Granian's routine "Received close frame" / "Replying to close" DEBUG noise (paralleling the existing watchfiles entry); still visible at TRACE. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After a new deploy ships, browsers holding a cached index.html still reference the old hashed chunk filenames. The new server returns the SPA index page (text/html) for the missing /static/assets/*.js, and the dynamic import fails with a MIME-type error. The reader (or any lazy-loaded route) silently fails to open until the user force- refreshes. Fix it on both ends: - frontend: catch chunk-load errors in router.onError and force a full-page navigation to the target path. A sessionStorage guard prevents reload loops if the second fetch also fails. - backend: emit Cache-Control: no-cache on the SPA shell from IndexView so browsers always revalidate the chunk references. Hashed /static/assets/* files keep their immutable cache via servestatic. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r click (#705) * metadata: drop redundant structuredClone in getMetadata The previous implementation cloned ``{ ...rawSettings, filters }`` via ``structuredClone`` purely so it could ``delete data.mtime`` without mutating the caller's settings. But the filter payload comes straight from the Pinia browser store, which keeps the inner ``filters[k]`` arrays as Vue reactive Proxies. ``toRaw`` only unwraps one level, so those proxied arrays survived into ``structuredClone`` — which refuses to clone certain Vue proxies, surfacing as ``DataCloneError: [object Array] could not be cloned`` whenever a user clicked a Genre / Author / etc. chip in the metadata dialog. The clone was redundant in any case: ``serializeParams`` (the only consumer of ``data``) already deep-clones via ``_deepClone``, which calls ``toRaw`` at every level. Replace the clone-then-delete dance with a destructure that strips ``mtime`` cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * frontend: scrub remaining structuredClone(reactive) hazards The ``getMetadata`` ``DataCloneError`` fix landed earlier in this PR applied to one site; the same anti-pattern lived in five other places. Each one cloned a Vue reactive value (Pinia store row, breadcrumb, route params) via ``structuredClone(toRaw(x))`` — the outer ``toRaw`` unwraps only one level, so a nested reactive Array proxy can still trip ``DataCloneError: [object Array] could not be cloned``. Two cleanup categories: * **Destructure-and-drop** (breadcrumbs, reader close button) — the clone existed only to ``delete params.name``. Replaced with ``const { name: _name, ...params } = toRaw(crumb)``: no clone, no proxy hazard. * **Genuine deep copy** (admin create/update dialog + its mixin) — promoted the existing private ``_deepClone`` in ``api/v3/common.js`` to an exported ``deepClone``. It already does what these sites need: recursive ``toRaw`` at every level, no structuredClone fragility. Swapped four ``structuredClone(toRaw(x))`` calls for ``deepClone(x)``. Also dropped one defensive clone entirely (``reader-book-change-nav-button.vue``): vue-router snapshots ``params`` itself, so the outer clone was never load-bearing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* identifier: migration to clear stale Metron URLs that 404
Comicbox emitted metron.cloud/{genre,location,story,tag,role}/... URLs
for identifier types that have no public web pages on metron.cloud — only
API endpoints. Those links always 404. Drop them from existing identifier
rows so the metadata view stops surfacing broken links. Re-import will
repopulate where applicable (roles pick up the creator URL via comicbox's
computed-step fallback).
Pairs with comicbox PR #124 (ajslater/comicbox#124),
which stops generating these URLs at import time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* identifier: fold stale-Metron-URL cleanup into migration 0039
0039 is unreleased and already a multi-purpose data migration; tucking
the Identifier.url cleanup into it keeps the v1.11 migration count
unchanged. Adds _clear_stale_metron_urls() and runs it after the other
data backfills.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…vulture (#708) * lint: replace vulture with skylos Vulture's analysis pass was the slow link in the python lint pipeline. Skylos covers the same dead-code role and runs much faster. - Lint dep: vulture~=2.3 -> skylos~=4.10 - bin/lint-python.sh: invoke skylos --no-upload --confidence 61 (61 mirrors the previous vulture min_confidence and drops skylos's noise-floor bucket of _PREFIX module constants) - pyproject.toml: replace [tool.vulture] with [tool.skylos], including exclude list, [tool.skylos.masking].bases for Thread/Serializer/ ModelSerializer/BaseImporter framework patterns, and [tool.skylos.whitelist].names carrying the four prior vulture_ignorelist entries plus DRF Meta - Drop vulture_ignorelist.py and remove its references from basedpyright, complexipy, coverage, ruff, ty, and the wheel source-include list - Gitignore the .skylos/ cache dir Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * formatting * lint: prune dead code surfaced by skylos triage + tighten skylos config Removes the 12 dead-code items from the skylos investigation (tasks/skylos-triage.md), extends the skylos whitelist to suppress the recurring framework-pattern false positives, and switches admin library re-poll to fire only when pollEvery changes. Code removed (no callers, all internal): - LibraryPollerThread.wake() — redundant with poll(); never called - CoercingSmallIntegerField — only the Positive variant is used - Counts.search_changed (importer/init.py) - QueryPruneLinksFKs.pop_links_to_fts - QueryIsUpdateImporter._query_normalize_existing_values - SearchIndexSyncManyToManyImporter._to_fts_str - SearchIndexerRemove.remove_duplicate_records - ReaderSettingsBaseView._get_bookmark_auth_filter (refactor leftover; the inherited get_bookmark_auth_filter is the live one) - OPDS1FacetsView._is_facet_active (part of the abandoned facet:activeFacet feature; the neighbor's own comment said as much) - allowed_ratings_for / rating_index / _RATING_INDEX (cascaded — only allowed_ratings_for was directly used; the others followed) - text_choices_from_string - MetronAgeRatingChoices (age_rating is an FK — choices wiring not needed; the unused-helper-+-import is just clutter) Behavior change: - AdminLibraryViewSet.perform_update now only triggers _poll() when pollEvery is in validated_keys. Other field edits (groupSet, covers_only, …) are picked up by the next already-scheduled poll rather than forcing an immediate scan. skylos config: - bin/lint-python.sh: --category dead_code --no-provenance (37s -> 16s; report scoped to dead-code only) - [tool.skylos.whitelist].names extended for DB-router protocol, URL-converter protocol, validate_*, framework class attributes (lookup_field, default_code, view_is_async, …), Janitor task dispatch, importer phase dispatch, and OPDS2 _publication_* getattr dispatch — the patterns skylos's AST analysis can't see - tasks/skylos-triage.md: investigation report and remaining-noise inventory Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * lint: revert from skylos back to vulture for the regular lint pass The one-time skylos triage was useful (it surfaced the 12 dead-code items already cleaned up earlier on this branch), but as a recurring lint step skylos is too noisy: ~110 of its findings on this codebase are unfixable framework-pattern false positives — DRF metaclass field assignment, Django getattr-by-name protocol attributes, and string-name dispatch tables that its static analysis can't see. - pyproject.toml: vulture~=2.3 back in lint deps; [tool.skylos] block (~80 lines of whitelist + masking) replaced with the original [tool.vulture] block; vulture_ignorelist.py restored to the source-include / basedpyright / complexipy / coverage / ruff / ty exclude lists. - bin/lint-python.sh: skylos invocation -> vulture . - vulture_ignorelist.py: restored with the original four entries. - .gitignore: drop the .skylos/ cache entry. - tasks/skylos-triage.md: removed. The dead-code cleanups skylos surfaced earlier in the branch are kept as-is — they're real wins regardless of which tool found them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The comicbox alpha bump removed URLs from a couple of metron tag identifiers (genre/012 and story/555). Update the expected fixture values to match: URL field is now None for these identifiers in both AGGREGATED_UPDATE_ALL and QUERIED_UPDATE_ALL. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The earlier ``_granian`` filter (#704) didn't catch the close-frame DEBUG messages users were still seeing — those originate from the ``tungstenite`` and ``tokio_tungstenite`` crates that Granian uses for the WebSocket layer, not from ``_granian`` itself. pyo3-log routes Rust tracing targets to Python loggers, mapping ``::`` → ``.`` so e.g. ``tungstenite::protocol`` arrives as the Python logger ``tungstenite.protocol``. Filtering the bare crate name applies hierarchically to all submodules. Verified by applying the dictConfig and reading effective levels: ``tungstenite.protocol`` and ``tokio_tungstenite.compat`` now resolve to INFO with ``isEnabledFor(DEBUG) == False``; root stays at DEBUG so codex code is unaffected. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ``v-else-if`` / ``v-else`` chain on the page wrapper made ``LoadingPage`` and the image component mutually exclusive. When the 333ms ``mounted()`` timer flipped ``showProgress=true``, Vue swapped to ``LoadingPage`` — destroying the ``<img>`` element along with its ``@load`` listener. Any response arriving after that landed on a torn-down element, ``loaded`` never flipped, and the spinner stuck forever. Only reproduced in production: behind an nginx reverse proxy with cold caches, the first page-image request crossed 333ms; on ``make dev-prod-server`` against ``localhost`` the image came back well under the threshold so the swap never fired. Render ``<component>`` and ``LoadingPage`` as siblings under ``v-else`` with ``v-show`` toggling component visibility instead of ``v-if``. The component stays mounted across the spinner transition, the ``@load`` handler is still wired when the response lands, and the visible behaviour matches the previous template at each state (image during 0–333ms grace, spinner while loading beyond that, image once loaded, ErrorPage on error). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ack threshold (#712) Two fixes bundled, both surfacing as noisy WARNINGs in dev mode. ## 1) UserAuth invariant UserAuth (renamed from UserActive in 0039) was only provisioned by ``AdminUserViewSet.perform_create``. Anything else — ``manage.py createsuperuser``, fixtures, factory_boy — left the User without a matching row, so the bookmark thread's hourly activity touch fired "No UserAuth row for user pk=N; skipping touch." on every active user. The fix is a single invariant: every ``User`` has exactly one ``UserAuth``. Three pieces: - New ``post_save`` signal in :mod:`codex.signals.django_signals` ``get_or_create``s a default ``UserAuth`` for every new User, so every creation path now satisfies the invariant. - 0039 picks up a ``RunPython`` step that backfills a default row for any pre-existing User missing one (the legacy lazy-create ``UserActive`` gap, plus any ``createsuperuser``-provisioned admin). Folded into 0039 rather than a separate 0040 since the branch is still alpha. - ``perform_create`` and the related ``UserSerializer._apply_userauth`` rely on the row pre-existing now: ``perform_create`` only patches the admin-supplied ``age_rating_metron`` ceiling onto the signal-created row. Tests that explicitly created their own ``UserAuth`` row alongside a ``User`` were updated to either drop the redundant create or patch the auto-created row, and the ``MigrationShapeTestCase`` "empty table" assertion was flipped to assert the new invariant. The ``update_user_active`` warning intentionally stays — at this point a missing row really would mean a real data-integrity issue worth surfacing. ## 2) asyncio slow-callback threshold ``bin/dev-server.sh`` exports ``PYTHONDEVMODE=1`` when DEBUG=1, which flips asyncio into debug mode. The default ``slow_callback_duration`` of 100ms is calibrated for pure-async codebases — Codex routes most requests through asgiref's ``AsyncToSync`` (a sync Django view on a thread, awaited from the loop), and a DB-heavy view crossing 100ms is routine, not an anomaly. The warnings drown out genuine slow-task signal. Bump ``loop.slow_callback_duration`` to 5.0s in :func:`_serve` so the warning still catches truly slow callbacks (a 5s+ async operation is a real bug worth surfacing) without spamming on every normal sync-view request. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Django's base cache class checks every key for memcached compatibility (no spaces / control chars / length > 250) and emits a ``CacheKeyWarning`` for violators. Codex hashes-and-pickles to disk via the file-based backend; the FS layer accepts arbitrary keys, so the memcached portability warnings are pure noise. Cachalot composes keys from query plans that include tuples like ``(127, 128)`` (pk ranges from in-clauses), and those tuple repr spaces trip the warning on every browse / cover request — a couple of warnings per page-load in production. Override ``validate_key`` to a no-op. The corruption-tolerance docstring already explains why this backend exists; extend it to mention the validate_key skip. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* bump to version 1.11.0a5 * news tweaks * bump to 1.11.0a6 * picopt run changes comic.svg and comic-165.webp * update alpha version to 1.11.0a7 * update deps
ajslater
added a commit
that referenced
this pull request
May 4, 2026
* update deps
* v1.10.1 fix opds v2 links
* fix news
* fix gha trigger gate logic
* fix gha yaml bug
* fix gha build and deploy from firing on develop
* fix admin update user view & serializer to allow user updates.
* fix gha yaml syntax error'
* fix runing build on develop branch
* fix gha branch gate logic again
* skip tests on deploy if identical files. change discord colors & messages. move ci contaiiner to compose
* fix polling all libraries when no ids submitted in task
* fix poll all library button
* fix opds v2 manifest series link
* force browser reset on start page of opds v2
* fix dev csp for vite hmr
* working server folder picker with enter activates menu and waits to render so the menu isn't lost in the center of the screen
* clean up server folder code to be smaller and remove cruft
* bump news
* attempt to make granian auto reload not hold on to file handles by adjusting template settings in debug mode
* Squashed commit of the following:
commit c76660006840abb36aa37d7355d5c7e242babebf
Merge: b2b5a011a be94a0ae0
Author: AJ Slater <aj@slater.net>
Date: Sun Apr 5 15:49:11 2026 -0700
Merge branch 'develop' into select-multiple
commit b2b5a011ab207a7307505b092877f789e1a66ab3
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:15:54 2026 -0700
fix card menu hover highlighting
commit 5a98fe23d89be5e11fa46ada927100ccd3e08cda
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:13:37 2026 -0700
new design for select many mode. no settings drawer involvement. reconfigure cards
commit 82c612e3f77eec90b02465174b49fc8552871686
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:50:11 2026 -0700
add github config to source include. remove dockerhub config
commit b02bd450f4598300d78433a2d749741872d14894
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:42:32 2026 -0700
update deps
commit d7bbc875e7936d808259c9ba726127fcc3af905e
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 23:56:12 2026 -0700
select many toolbar
commit 246d5ecda59aaabb83e41db3ad32ec552804b9ae
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 22:23:24 2026 -0700
select many feature first attempt
* bump news
* efficiency and bugfixes for gha yaml
* fix container name for gha
* fix test report steps running if tests don't run
* use eslint_d for frontend lint
* make sure to build the ci compose locally
* fix too exuberant volumes mounting for gha compose
* remove working dir entry i'm not sure it's good
* add debug line to gha workflow
* remove debug line
* fix test make order
* stop docker compose when done in gha
* i don't think i need DJGNO_SETTINGS_MODULE th ci thing
* fix down call to own bring down ci task
* fix docker compose down"
* fix test results upload
* fix test step name
* fix gha permissions
* bump news
* update deps
* browser set_params method saves last route and saves params to settings
* bump version to 1.10.4
* modify gha discord notification
* use happy-dom for frontend tests
* docker tag script better than the one in ci/
* update deps
* far saner param initialization for browsers and opds start pages
* update dpes & bump alpha version
* v1.10.5a0 (#551)
* bump news
* cirlcle ci no longer handles pre-release
* remove alpha scripts for circleci"
* fix lint-ci
* fix pre-relase gha
* fix default last route on start pages
* bump version
* fix default params
* alpha2
* regular version 1.10.5
* remove develop circleci builds
* adjust variable name
* bump version to 6 alpha 0
* fix default params for feed_views in opds 2
* debug logging for django crash
* v1.10.6a0 (#553)
* bump news
* ignore ruff qa for debug build
* ignore shellcheck"
* lint
* minor refactors of opds v2 feed
* regular v1.10.6 version
* fix docker-tag-latest cscript
* ignore gh token file
* try to log more request errors
* reconfigure logging to hopefully be more verbose about request errors in production
* update deps and bump to alpha version
* bump news (#555)
* v1.10.7
* bump news
* update devenv
* update devevn and deps
* fix pm script
* move django-check to test category
* workflow build frontend and collect static for prodcution build. fix test upload
* bump version and news 1.10.8
* fix dev-module script
* fix news
* explain news
* fix opds clear search setting
* fix clear search button
* use a registry cache instead of gha cache for the dist-builder
* gha use more env vars for image names. retain python dist for 2 days.
* new quick deploy gha script. update deps & devenv.
* silence watchfiles 5 second timeout debug message
* consolidate null values const
* make scope private
* update devenv
* update deps. migrate to unhead v3
* update deps
* fix creating reader global settings
* fix caching
* rename codex build-dist to codex-ci
* fix image name. make gha steps depend on each other more.
* fix gha syntax errors
* names for gha steps
* use ghcr.io for python-debian base
* update deps
* format dockerfile
* picopt treestamps
* fix custom covers not importing. v1.10.11
* fix custom covers count in admin view
* bump news for custom cover count fix
* update deps
* codex identification in server tag and opds generator tag
* update deps
* force no entries on opds start page
* common opds start page mixin. emtpy group objects on start page
* update deps. typechecking.
* api change q to search
* standardize search param as 'search' instead of 'q' or other variations
* remove errant icecream
* clear settings on backend
* Squashed commit of the following:
Fix clear settings null bug, add global settings clear button
- clearComicSettings was setting book.settings to null, causing
Object.entries() to throw TypeError downstream in getBookSettings
- Add null guard in getBookSettings as defense-in-depth
- Add null guard in isClearDisabled computed
- Add clearGlobalSettings action and clear button to Default Settings panel
- Compare against READER_DEFAULTS to determine if global clear is disabled
* simplify settings class hierarchy
* rename select-many store to browser-select-many
* switch to bun. updated devenv
* add a claude md
* use frozenattrdict to speed up configuration
* fix rename of browserSelectMany store
* auth token help
* fix sort-ignores to make deterministic across shells with different locales
* fix crash on settings not being raw
* another gaurd for getMetadta()
* remove keys from unhead meta headers
* fix unhead description for admin tabs"
* fix overzealous lazy importer
* fix lazyImportEnabled variable in metadata-activator
* fix metadata activator from bad cherry pick
* fix errant quote
* fix typechecking
* update devenv & deps
* bump news
* fix import bug linking folders
* fix possible batching crashes. adjust import variables for throughput
* batch comic updates
* move INTERNAL_IPS setting to general django area
* fix typechecking issue
* update deps
* bump version to v1.10.12
* fix redirect on OPDS alternate view with metadata
* minor change to browser empty page for better first time experience
* allow browsing to comics with any top group in opds
* bring project up to speed with bun and no package-lock.json
* fix browser paginator
* bump news for browser paginaor fix
* fix search combobox clearing
* uppercase book close button
* version v1.10.13. update deps
* fix pdfs not displaying
* fix csp for pdfs
* fix OPDS FK constraint failure when session row is missing (#607)
iOS Panels (and other Basic-Auth OPDS clients) intermittently hit
sqlite3.IntegrityError: FOREIGN KEY constraint failed when settings
or bookmarks were saved. Two interacting bugs caused it:
- Janitor cleanup_sessions used `if not session.get_decoded():` to
detect "corrupt" sessions. get_decoded() returns {} for both real
decode failures and legitimate anonymous sessions with no stored
data — exactly what Basic-Auth OPDS clients produce. The nightly
task was wiping valid session rows. Replaced with a direct
signing.loads() call so only genuine signature/decode failures are
flagged.
- _ensure_session_key returned the cookie's session_key without
verifying the row still exists. With cached_db the session loads
from cache without rechecking, so a stale cookie key would slip
through and cause an FK violation when used as SettingsBrowser /
SettingsReader.session_id. Now we verify existence and flush+save
to cycle the key when the row is gone.
Either fix alone closes the user-visible error; both together also
stop the underlying churn that created the bad state.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* extend session-key validation to bookmark + reader-settings paths (#608)
The same stale-session_key FK-violation pattern existed in two more
places that also write rows whose session FK can be stale:
- BookmarkAuthMixin.get_bookmark_auth_filter — feeds session_id into
Bookmark.objects.bulk_create / bulk_update.
- ReaderSettingsBaseView._get_bookmark_auth_filter — feeds session_id
into SettingsReader.objects.create.
Both used the old `if not session.session_key: save()` pattern that
trusts the cookie. Hoist the validated _ensure_session_key helper
from SettingsBaseView up to AuthMixin so every auth-aware view shares
one implementation, and switch both call sites to it. BookmarkAuthMixin
now extends AuthMixin to inherit the helper. BookmarkFilterMixin is
unchanged — it's read-only (filter Q only) and a missing session
correctly returns no rows.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* format
* bump news
* Stats: fix user_registered_count / auth_group_count always-zero bug (#611)
The /admin/stats endpoint's user_registered_count and
auth_group_count fields have been silently returning 0 since at
least Sep 2024. The Stats tab in the admin UI shows 0 registered
users even on installs with multiple accounts.
Root cause: _add_config tried to rename the per-model count keys
produced by _get_model_counts:
config["user_registered_count"] = config.pop("users_count", 0)
config["auth_group_count"] = config.pop("groups_count", 0)
But _get_model_counts builds keys via
``snakecase(model.__name__) + "_count"``. For Django's
``django.contrib.auth.models.User`` / ``Group`` that produces
``user_count`` and ``group_count`` (singular). The pop()s with the
plural names never matched, so the default ``0`` won every time —
and the actual ``user_count`` / ``group_count`` keys were left
orphaned in the dict, then dropped by the StatsConfigSerializer
which only declares ``user_registered_count`` /
``auth_group_count``.
Fix: pop the right source keys.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* defaults for dockerfile ARGs
* format'
* Frontend correctness: 10 bug fixes from sub-plan 01 (#647)
* Reader page: fix load-progress spinner that never appeared
Two stacked bugs in the same setTimeout:
1. Non-arrow callback lost ``this``. The function ran with the
timer's context, not the component's, so ``this.loaded`` and
the write below were both no-ops.
2. The write targeted ``this.loading``, which has never been a
data field on this component. The template binds the spinner
to ``showProgress`` (line 15: ``v-if="showProgress && !loaded"``).
So even if the arrow had been there from the start, the spinner
still wouldn't have rendered — both bugs had to land at once.
Net: ``LoadingPage`` has been dead code for slow image loads.
Switch to an arrow function and write ``showProgress`` instead
of ``loading``. Stash the timer ID so ``beforeUnmount`` can
clear it; a fast page swap mid-delay would otherwise fire the
write on a torn-down component.
Implements B1 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: fix arc-mtime fallback that itself 500'd
``loadMtimes`` builds an arcs list of ``{ group, pks }`` from
``this.arcs``; if the dict is empty the function previously
fell back to ``arcs.push({ r: "0" })``. The comment noted that
"No arcs is a 500 from the mtime api" — the fallback was added
to dodge that 500 — but the wrong-shape fallback also produced
a 500 because the API expects ``group``/``pks`` keys, not ``r``.
Use the canonical shape so the fallback actually works.
Implements B2 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* iOS PWA download: pass the object URL to revokeObjectURL, not the Blob
``URL.createObjectURL(blob)`` returns a ``blob:...`` URL string;
``URL.revokeObjectURL`` must receive that same string to free the
mapping. The previous code passed ``response.data`` (the Blob
itself), which silently no-op'd and leaked one object URL per
download.
On iOS PWAs this matters more than elsewhere because the leak
accumulates across the user's session and can't be reclaimed
short of reloading the app.
Implements B6 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser API: stop mutating caller's settings in getGroupDownloadURL
``getGroupDownloadURL`` did ``delete settings.show`` on the
caller's object before building the URL. Side-effect: any caller
that re-used the settings dict after the download-URL build saw
its ``show`` key silently vanish. This was probably fine when
the function was first written but it's a footgun now that
settings flow through a Pinia store.
Destructure-and-spread to drop ``show`` without touching the
input.
Implements B8 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Auth store: make logout awaitable + clear state unconditionally
Two changes to the logout action:
1. ``async`` so callers can ``await``. The current call site
(``auth-menu.vue``) fires and forgets, but a future UX pass
that wants to disable the button while logout is in flight
needs the promise.
2. Clear ``this.user`` in ``finally`` rather than only on
success. The user clicked "log out" — UI should reflect the
logged-out state immediately, regardless of whether the
server-side logout endpoint succeeded. Server-side cookies
that survive the network failure will get cleaned up by the
next 401.
Implements B7 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser filter menu: stop double-rendering each filter row
``<v-list>`` was passed both ``:items="vuetifyItems"`` AND a
default-slot ``v-for`` over the same list. Vuetify renders the
items prop into ``v-list-item`` children directly, so every row
was being built twice — once by the prop, once by the manual
``v-for``. Visible to users on filter menus with large choice
lists (genres, characters, etc.); each row appeared duplicated
and the DOM cost doubled.
Drop the prop. Keep the ``v-for`` because it carries the custom
``#append`` slot for ``metronName`` rendering. ``:model-value`` /
``@update:selected`` still drive selection state via each list-
item's ``:value`` prop.
Implements B10 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Admin job-tab: remove API-fetching click handler from expanded panel
The expanded status panel had ``@click="loadAllStatuses"`` on its
container div. Any click inside the panel — including clicks on
child elements that bubbled — refetched the entire status map.
Probably copy-pasted as a "refresh on click" gesture, but it
fired far too often: a user inspecting a long status list would
trigger N API calls just from glancing around.
The status data is pushed through the websocket already
(socket.js fans librarian notifications into the admin store),
so the panel is up-to-date without a manual refresh.
Implements B11 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: handle bookmark-write errors instead of silently rejecting
``setRoutesAndBookmarkPage`` awaited ``_setBookmarkPage`` but
didn't catch its errors. On a network blip the promise rejected,
the bookmark didn't persist, and the failure became an unhandled
rejection in the browser console — not visible to the user, not
retried, just lost.
Wrap in try/catch. The local page state stays where it is (the
user is reading forward; the bookmark catches up on the next
write), but the failure is logged so debugging surfaces. A
proper user-visible toast + retry path is broader UX work
tracked in the plan.
Implements B3 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: clear progress timer on unmount
``updateProgress`` chains itself via ``setTimeout`` until the
metadata loads or progress reaches 100. The timer ID was never
stashed, so closing the dialog mid-animation left the chain
running — each tick fired on a torn-down component, writing
``this.progress`` and re-scheduling against now-null refs.
Stash the timer ID and clear it in ``beforeUnmount`` so the
chain stops cleanly when the dialog goes away.
Implements B12 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader pager: key dynamic component on identity to force remount
``<component :is="...">`` without ``:key`` lets Vue reuse the
existing instance across an ``is`` change when the components
share enough surface (props, name). For the reader's
vertical/horizontal pager swap that's wrong: scroll listeners
attached by the previous mode persist, the new mode's
``mounted`` runs against stale internal state, and any
abort/teardown logic in ``beforeUnmount`` never fires.
Add ``:key="component.name"`` so the swap is a true unmount +
remount — old listeners go away, new mode starts clean.
Implements B13 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: lint cleanup for the B12 fixup
Vue option-order rule: ``beforeUnmount`` belongs above
``methods``. Block-comment style required for the multi-line
explanation in ``updateProgress``. Both surfaced when running
eslint on the prior commit; pure cleanup, no behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* news for cherry pick
* OPDS auth: send WWW-Authenticate + opds-authentication content-type on 401 (#652)
Panels and other strict OPDS clients require the WWW-Authenticate header
(per RFC 7235) to trigger the auth prompt, and the spec calls for
application/opds-authentication+json on the auth document. The exception
handler was already converting 403 to 401 with the auth doc body, but
was bypassing DRF's natural 401 response and dropping both the header
and the proper content type, which Panels read as a forbidden state.
Also fix a stray `from re import DEBUG` in the auth view that always
forced the absolute-URL path.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix typing inheritence with drf perms
* bump version to v1.10.14
* update picopt treestamps
* fix typing import error
* fix vulture ignorelist
* readd the mistakenly removed vulture_ignorelist
* v1.10.15 fix show order bug. update deps
* fix nightly job manual expansion
* fix news version number
* V1.11 performance (#714)
* title for age rating tagged
* rename unrestricted to adult for clarity
* remove circleci code
* update to new devenv scripts in frontend
* fix sort-ignores to make deterministic across shells with different locales
* batch comic updates
* fix possible batching crashes. adjust import variables for throughput
* bump news for v1.11.0
* Add features to readme
* add saved view feature
* Add browser-views perf measurement harness (Stage 0) (#574)
Stand up the measurement infrastructure that gates the rest of the
browser-views performance work:
- Add django-silk 5.5 to the dev group; install + route it to a
separate ``silky`` SQLite DB via a new ``SilkRouter`` so profiler
traces never touch the live DB.
- Wire SilkyMiddleware below ServeStaticMiddleware so it only wraps
the API stack, not static-file responses.
- Expose /silk/ under the existing DEBUG URL block.
- Add ``tests/perf/run_baseline.py``: hits the live dev DB via
``django.test.Client``, runs three cold/warm flow pairs (root
browse, filtered search, series metadata), and writes a JSON
baseline artifact. Cachalot + page-cache are invalidated before
each cold pass so the numbers reflect actual DB work.
- Add ``make perf-baseline`` target.
- Commit the per-view analysis docs and initial baseline capture
under ``tasks/browser-views-perf/`` for reference during the
follow-on cleanup stages.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 1 - startup + annotation + filter micros (#575)
Bundle of small, surgical wins observed on the slimlib baseline:
- Cache `libraries_exist` with a 60s TTL in Django's default cache;
invalidate on Library save/delete via lazy-imported signal handlers.
Drops a redundant `Library.objects.filter(...).exists()` from every
browser response.
- Short-circuit the four `_save_browser_*` code paths in
`codex/views/settings.py` when nothing actually changed. Avoids
gratuitous `.save()` calls that flush cachalot on no-op PATCH writes.
- Dedupe the `add_group_by(qs)` call in
`codex/views/browser/browser.py`: group once in
`_get_common_queryset` (no-op for Comic), drop the second call in
`_get_group_queryset`.
- Memoize `get_max_bookmark_updated_at_aggregate` per
`(model, agg_func, default)` on the view instance — the three
callers (group_mtime, order, bookmark) now share one Aggregate.
- Move the `bmua_is_max` flag off the per-row `Value` annotation and
read it from `self.context["view"]` in the browser serializer.
- `@lru_cache(maxsize=256)` on `_preparse_search_query`: extract to a
pure module-level helper keyed on `(text, path_allowed)`; returns
frozen tokens.
- `@lru_cache(maxsize=512)` on `get_field_query`: cache the Lark-parsed
Q tree, `copy.deepcopy` on return because `_hoist_filters` mutates
`child.negated` downstream.
- Stash the `BaseDatabaseOperations(None)` singleton as `_DB_OPS` in
`search/field/expression.py`; `prep_for_like_query` doesn't use the
connection.
- Pre-filter the field loop in `filters/field.py` to keys actually set
in the request — saves ~20 no-op calls per browser.
- Delete `search/field/optimize.py` (`like_qs_to_regex_q` and friends
were already unused — grep confirms only self-references).
Slimlib cold baseline (3 flows, stage1.json vs Stage 0 baseline.json):
- flow_a_root_browse: 21→18 SQL, 189.6→182.3 ms
- flow_b_filtered_search: 21→17 SQL, 187.4→178.0 ms
- flow_c_series_metadata: 34→31 SQL, 251.1→226.9 ms
Warm paths unchanged (0 SQL, ~2 ms). Full tests + ruff pass.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix type warnings
* fix lint error
* format
* Run silk migrations on silky DB at startup (#576)
Stage 0 wired a second SQLite DB (silky) for django-silk profiling traces
and a DATABASE_ROUTERS entry that routes silk app models there. But
ensure_db_schema only invoked `call_command("migrate")` without a database
arg, which only migrates the default DB. The router then blocks silk
migrations from running on default — so the silky DB was never populated,
and the first request through SilkyMiddleware failed with
"no such table: silk_request".
Mirror the pattern from tests/perf/run_baseline.py: after the default
migrate, also run `migrate silk --database=silky` when the silky DB is
configured. Guarded on `"silky" in connections.databases` so production
(DEBUG=False, no silky DB) is a no-op.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 2 — triple COUNT + page mtime cache (#577)
## PR 2a — Eliminate triple COUNT on the paginate path
Each browse request ran three COUNT queries per section (groups & books):
1. The outer grouped COUNT in `_get_common_queryset`.
2. Paginator's internal COUNT triggered by `paginator.page()`.
3. An explicit `.count()` on the paginated slice.
The outer COUNT is needed (sizing the paginator). The other two are
redundant — the page row count is bounded by `per_page` and derivable
from `end_index - start_index + 1`.
- Shadow `Paginator.count` (a `@cached_property`) with the pre-computed
total to skip Paginator's internal COUNT.
- Derive page row count arithmetically from `Page.start_index()` /
`end_index()`.
- Pass `book_count` through `paginate()` alongside `group_count`; drop
`book_qs.count()` on the opds2 path.
- `_paginate_section` returns `(qs, count)` directly.
Short-circuits on `total_count == 0` (avoids Paginator instantiation on
empty sections) and preserves the EmptyPage warning branch.
## PR 2b — Short-TTL page mtime cache
`BrowserView._get_page_mtime()` calls `get_group_mtime(page_mtime=True)`
on every browse request. The query is a filtered Max aggregate that
cachalot caches — but any write to Comic / Bookmark invalidates it, so
bookmark-heavy usage forces recomputation. Cold-path silk traces show
this aggregate at ~26ms on flow_a — the second-slowest query in the
request.
Add a 5s TTL cache layer gated on page_mtime=True. Key includes user,
model, group, pks, page, and a hash of filter-affecting params
(filters, search, q, order_by, order_reverse). The polling MtimeView
path (no page_mtime) is unaffected, so frontend change-detection stays
live.
## Measurements
tests/perf/run_baseline.py on the slimlib DB. Cold = full cache
invalidation; warm = cachalot populated.
| flow | stage1 cold | stage2 cold |
|-------------------------|------------------------|------------------------|
| flow_a root browse | 18 queries / 182.3 ms | 16 queries / 135.6 ms |
| flow_b filtered search | 17 queries / 178.0 ms | 15 queries / 130.3 ms |
| flow_c series metadata | 31 queries / 226.9 ms | 31 queries / 229.9 ms |
flow_a / flow_b: -2 queries, ~26% cold wall-time reduction. flow_c
unaffected (metadata doesn't traverse paginate). PR 2b's benefit is a
dogpile guard after cachalot invalidation — doesn't show in the harness
(cold = both caches empty; warm = cachalot wins first).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix browser paginator
* typecheck browser paginate
* Browser views perf: Stage 3 — cover fan-out collapse (#578)
* Stage 3 — cover fan-out: per-pk endpoint + pre-resolved cover_pk
Before stage 3, every cover card triggered a full BrowserAnnotateOrderView
pipeline to pick the representative comic pk. A default page does ~100 of
these in parallel. This PR collapses that into one correlated Subquery on
the browse response plus a thin per-pk cover endpoint.
Components
- Option A: annotate_order_aggregates(for_cover=True) drops the JsonGroupArray
+ page_count aggregates from the cover path — unused for picking a pk.
- Option B.1: BrowserView group cards get cover_pk / cover_custom_pk via
correlated Subquery that replicates CoverView.get_group_filter exactly
(direct fk match when dynamic_covers/Volume/Folder; sort_name fuzzy match
otherwise, correlated on _GROUP_BY columns so the same comic set as `ids`
is picked — no JSON parsing, no peer aggregate).
- New endpoints /api/v3/c/<pk>/cover.webp and /api/v3/cc/<pk>/cover.webp
serve already-resolved pks with a cheap single-row ACL probe and the
existing CoverPathMixin / CoverCreateThread pipeline.
- Frontend getCoverSrc prefers the new per-pk URL when cover_pk /
cover_custom_pk is on the card; falls back to the old group+pks URL
otherwise, so OPDS and search-active browses keep working.
- FTS skip: cover_pk annotation is skipped when params["search"] is set.
MATCH inside a correlated subquery re-scans the FTS5 index per outer
row (~900ms on a 100-group page). The old URL path still applies the
search filter per cover — same behavior, parallelized over HTTP.
Perf (slimlib dev DB, Flow D = browse + every card's cover):
Flow Before cold After cold Delta
A root browse 156.9ms / 15 q 180.5ms / 15 q +24ms / +0 q
B search 148.3ms / 16 q 132.4ms / 16 q -16ms / +0 q
C metadata 161.9ms / 31 q 165.0ms / 31 q +3ms / +0 q
D browse+covers 2311ms / 1216q 1161ms / 815 q -50% / -33%
Flow A takes a mild regression for the correlated cover subquery, but Flow
D — the realistic user wall-clock — drops by half and 400 SQL queries.
Subsequent cover fetches drop from ~90ms each (full pipeline) to ~5ms
(disk read + 1 ACL probe).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Stage 3 follow-ups: custom_cover URL, FTS pre-materialization, OPDS thin covers
Address review feedback on the cover fan-out collapse:
- Rename /api/v3/cc/<pk>/cover.webp → /api/v3/custom_cover/<pk>/cover.webp.
Descriptive over terse; matches the view name.
- FTS pre-materialization replaces the FTS-skip fallback. A correlated
MATCH re-scans FTS5 per outer row (~900ms on a 100-group page); the
old path worked around this by skipping cover_pk annotation on search
and sending every cover through the legacy pipeline once. We now
pre-select the FTS match set as a non-correlated sub-SELECT in the
outer cover subquery — SQLite materializes it once and each correlated
cover row lookup becomes an indexed pk filter. Cover_pk is annotated
on search responses, the thin endpoint handles each cover.
- OPDS now emits thin-endpoint cover URLs (v1 and v2):
- New OPDSComicCoverByPkView / OPDSCustomCoverByPkView wrap the browser
thin views with OPDSAuthMixin's Basic Auth.
- New opds:bin:cover_by_pk and opds:bin:custom_cover_by_pk URL names.
- v1 _cover_link picks: cover_custom_pk → custom thin; group=='c' → own
pk thin; cover_pk → thin; else legacy group+pks fallback.
- v2 _thumb always uses the thin endpoint (publications are Comic rows
so obj.pk IS the cover pk).
- OPDS inherits annotate_cover() via BrowserView._get_group_and_books,
so group rows already carry cover_pk / cover_custom_pk.
- tests/perf/run_baseline.py gains a flow_e (search + covers) to measure
the search path end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix search combobox clearing
* format
* Stage 3 follow-ups: folder covers, endpoint rename, offline cover pipeline (#579)
- Fix folder cover_pk picking comics not actually in a folder's subtree
by using the recursive ``folders`` M2M (same relation the browse filter
uses) for the per-card cover subquery instead of the direct
``parent_folder`` FK.
- Delete the legacy thick cover endpoint. Rename cover_by_pk.py to
cover.py and drop the "_by_pk" suffix from view class names
(CoverView, CustomCoverView, OPDSCoverView, OPDSCustomCoverView).
Update URL configs, OPDS wrappers, and the frontend client.
- Move all cover generation off the HTTP path. Add CoverCreateTask and
enqueue it from the importer after bulk_create so new comics get
thumbnails pre-warmed offline. When a cached thumb is missing the
per-pk endpoint enqueues the task and responds 202 Accepted with
Retry-After plus the missing-cover placeholder instead of synthesizing
the WebP inline under a worker thread.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update devenv sort ignores
* Force cover refresh after 202 so the placeholder doesn't stick (#580)
Browsers don't honor Retry-After on <img> elements and happily cache the
placeholder served with the 202 response, so even after the cover thread
finishes writing the real thumb the img src keeps rendering the stale
placeholder bytes.
- Backend sends Cache-Control: no-store on the 202 placeholder so the
response isn't cached at that URL.
- BookCover now probes the cover URL with fetch() on mount and, if it
sees 202, waits Retry-After and bumps a reactive `retry` counter that
becomes a cache-busting query param on coverSrc. v-img re-fetches with
the new URL and gets the real cover once the cover thread is done.
Retries are capped at 5 and aborted on unmount via AbortController.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Harden cover endpoint against concurrent read/write races (#581)
Under a full cold-cache page load, 32 HTTP workers hitting the cover
endpoint at the same time as the cover thread was writing thumbs
returned 500s for a handful of covers. The endpoint used a three-step
exists()/stat()/read_bytes() sequence that could race against an
in-flight write, and save_cover_to_cache wrote directly to the target
path so readers could observe a truncated file mid-write.
- save_cover_to_cache now stages to a ``{name}.{pid}.tmp`` sibling and
renames with Path.replace, so a read either sees the pre-existing
state or the fully written file — never a partial one. Stale temps
are unlinked on failure.
- _get_cover_response collapses to a single read_bytes() that catches
FileNotFoundError and logs OSError, then falls through to the 202
path. No race window between stat and read.
- LIBRARIAN_QUEUE.put and the outer get() methods are wrapped in
try/except that log and return the placeholder, so a surprise
exception is never user-visible as a 500 and always leaves a log line.
- cleanup_orphan_covers runs a dedicated ``*.tmp`` sweep over both cover
roots before the orphan scan, so stale temps from crashed writers
don't accumulate.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix custom cover generation
* relegate watch for changes restart to a en env variable settings
* await and finish watch task
* update devenv
* Fix metadata cover fetching pk=0 after Stage 3 cover fan-out (#583)
Stage 3's follow-up commit (2ff08524) dropped the legacy `group+pks`
fallback from the frontend `getCoverSrc()` helper, leaving it to only
build per-pk URLs from `coverPk` / `coverCustomPk`. The browser card
pipeline annotates those fields via `BrowserAnnotateCoverView`, but the
metadata endpoint was never wired into that annotation, and
`metadata-cover.vue` never passed the cover pks through to `BookCover`.
As a result every metadata dialog tried to load
`/api/v3/c/0/cover.webp` — pk 0 — and fell back to the missing-cover
placeholder.
Wire the annotation + serializer + component end-to-end:
- `MetadataAnnotateView` inherits from `BrowserAnnotateCoverView` so
`annotate_cover()` is available on the metadata queryset.
- `MetadataView.get_object()` calls `annotate_cover(qs)` after
`annotate_card_aggregates(qs)`, mirroring `browser.py`'s ordering.
- `MetadataSerializer` exposes `cover_pk` and `cover_custom_pk` as
`SerializerMethodField`s with the same fallback-to-`obj.pk` semantics
that `BrowserCardSerializer` uses — so Comic metadata (`group=c`)
works without annotation, falling back to its own pk.
- `metadata-cover.vue` forwards `md.coverPk` / `md.coverCustomPk` to
`BookCover`.
Verified on /api/v3/{p,i,s,v,c,f}/*/metadata: `coverPk` now resolves
to a real comic pk on every group, and the comic path falls back to
its own pk. No SQL query delta — `annotate_cover` is a correlated
Subquery that inlines into the outer SELECT.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hints (#582)
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hydration hints
Fold the three `_intersection_annotate()` passes (value fields, conflict
shadow fields, related-count fields) into a single batched call that
collapses N distinct-count probes into one `aggregate()` and one
`values_list()`, with unique synthetic keys so annotation groups don't
collide.
Extend `FK_QUERY_OPTIMIZERS` so intersecting FK hydration
(`AgeRating`, `Character`, `Team`) carries `select_related` + `.only()`
hints for the nested fields the metadata serializer actually reads
(`AgeRating.metron`, `Character.identifier.url`). Without this, each
nested access fired a follow-up query per instance.
For the M2M intersection path, short-circuit empty-intersection fields
with `Model.objects.none()` so we skip the optimizer setup (and, for
the Comic self-reference path, pointless prefetch dispatches on already
empty results).
Add `flow_c2_comic_metadata` to the perf baseline harness so the
comic-detail metadata path (the FK + M2M heavy flow) is tracked
alongside the series flow.
Measured: flow_c_series_metadata cold SQL drops from 31 → 28 (-3).
flow_c2_comic_metadata baseline captured at 47 queries for future
stages. Other flows unchanged within noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* WATCH for changes variable depends on debug
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Fix folder + FTS crash and preserve rank-ordered cover selection (#584)
* Fix folder browse + FTS crash on search_score ordering
Browsing folders (or any group) with an FTS search and
``orderBy=search_score`` blew up with::
sqlite3.OperationalError: no such column: codex_comicfts.rank
Root cause: Stage 3's cover fan-out collapse builds a correlated
Comic subquery per group card to pick ``cover_pk``. The follow-up
(#579) replaced the correlated ``comicfts__match`` filter with a
non-correlated ``pk__in fts_sq`` pre-materialization — so the Comic
subquery no longer joins ``codex_comicfts``. But
``annotate_order_aggregates`` still annotated
``search_score=ComicFTSRank()`` inside the subquery, and
``add_order_by`` issued ``ORDER BY "codex_comicfts"."rank" * -1``,
which SQLite can't resolve from the subquery's FROM list.
- ``_annotate_search_scores`` now takes ``for_cover`` and skips the
annotation when set. ``annotate_order_aggregates`` threads the flag
through (matching the existing ``for_cover`` pipeline-trims).
- ``_cover_comic_subquery`` passes ``order_key="sort_name"`` to
``add_order_by`` whenever the user's order is ``search_score`` — the
cover's tie-break isn't user-visible, and ``sort_name`` is a real
indexed column on Comic.
Verified: /api/v3/{r,f,a,p}/0/1?q=iron+man&orderBy=search_score all
return 200 with a real ``coverPk`` (not 0). Lint + 20-test pytest
suite pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Preserve FTS rank ordering in cover subquery
The previous fix short-circuited to ``sort_name`` when ``order_by`` was
``search_score`` so a group card's cover could be picked without
referencing ``codex_comicfts.rank``. That fixed the crash but meant the
cover chosen for each card was the alphabetically-first matching comic,
not the top-ranked FTS match — a UX regression on searched pages.
Restore rank-ordered cover selection by:
* Applying ``fts_q`` (``comicfts__match=...``) directly in the cover
subquery so ``codex_comicfts`` is joined and ``rank`` is populated,
while keeping the ``pk__in`` pre-materialization for cheap filtering.
* Teaching ``ComicFTSRank`` to resolve the query-local alias for
``codex_comicfts`` at compile time. Its literal template worked for
the top-level browse query but emitted an unresolvable column ref in
nested subqueries where Django aliases the join as ``V4``/``U1``.
* Skipping ``.group_by("id")`` in the cover-subquery search_score
annotation. The custom force-group-by compiler emits a literal
``"codex_comic"."id"`` that breaks under nested aliasing, and the
cover subquery is already ``.distinct() … LIMIT 1`` so dedup is
redundant there.
Verified against the dev DB: for each publisher card on a ``batman``
search, the annotated ``coverPk`` now matches the top-ranked comic
returned by a direct ``ComicFTSRank`` ordering within that publisher.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update for claude rules
* Browser views perf: Stage 5a — server-side cache_page on cover endpoints (#585)
Wrap /api/v3/c/<pk>/cover.webp, /api/v3/custom_cover/<pk>/cover.webp,
and their OPDS counterparts in cache_page(COVER_MAX_AGE). Compose with
cache_control + vary_on_cookie (API) or vary_on_headers("Cookie",
"Authorization") (OPDS, which accepts Basic + Bearer + Session auth)
so the Vary header is set before cache_page stores the response — the
cache key is keyed per auth identity, no cross-user leakage.
Also raise Django FileBasedCache MAX_ENTRIES from the default 300 to
10000. Cachalot query results + cache_page entries (browser + cover)
exceed 300 during a single browse-with-covers pageload, triggering the
2/3 random cull that silently evicts just-populated cover entries
before the next request can read them. Without this, Flow D warm only
dropped to 743 (~50% cover-cache hit rate); with it, Flow D warm drops
to 0.
Perf impact (stage5a-after.json vs. stage4-after.json):
Flow D — browse + 100 covers warm 802 → 0 queries
Flow E — search + 46 covers warm 368 → 232 queries
Flow E's residual 232 queries are 29 covers that return 202 Accepted
(cover not yet generated; response has Cache-Control: no-store, which
cache_page correctly skips). The 17 covers that returned 200 were 0
queries each. In production, 202s resolve within seconds as the cover
thread catches up, so steady-state warm on Flow E is also 0.
Cold query counts and Flow A/B/C are unchanged.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Raise RLIMIT_NOFILE on startup to avoid macOS 256 FD cap (#586)
Cold browser sessions loading a full 100-card grid intermittently crash
the dev server with `OSError: [Errno 24] Too many open files`. Diagnosis:
- macOS ships a 256 soft cap for RLIMIT_NOFILE (`ulimit -Sn 256`).
- Each Django request thread keeps a sticky SQLite connection
(`CONN_MAX_AGE=600`).
- SQLite WAL mode opens 3 FDs per connection (main + `-wal` + `-shm`).
- The thin cover endpoint dispatches to a `sync_to_async` threadpool, so
a burst of 100 cold cover requests easily spawns ~100 threads → ~300
SQLite FDs, blowing past the 256 cap before page reads even open.
Bumping the soft limit toward the hard cap (or 8192, whichever is lower)
at process start is non-invasive and matches what production deployments
typically achieve via shell `ulimit`. No-op on Linux (already high) and
on platforms without the `resource` module.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps, format json
* Browser views perf: Stage 5b — annotation gating + m2m-aware distinct (#587)
Three surgical correctness/perf changes that surface the right SQL for the
right target. The Stage 5a work (cover endpoint cache_page) collapsed Flow D
warm to 0; 5b cleans up cold-path waste the harness flows don't directly
exercise.
5.2 — Skip ``updated_ats`` JsonGroupArray outside browser/metadata
``obj.updated_ats`` is consumed only in
``BrowserAggregateSerializerMixin.get_mtime`` (browser + metadata
serializers). OPDS computes its own mtime from the bookmark aggregate;
cover/download paths never read it. Skip the DISTINCT scalar aggregate
for those targets.
5.3 — m2m-aware .distinct() in ``BrowserFilterView.get_filtered_queryset``
New ``comic_filter_uses_m2m`` cached property. Comic queries skip
``.distinct()`` unless a real m2m or m2m-through join is present
(story_arc browse, folder browse on cover/choices/bookmark/download
TARGETs, or any m2m field filter). Non-Comic queries still always
``.distinct()`` because the ACL alone traverses ``comic__``
(one-to-many).
5.5 — Tie ``search_score`` ``group_by("id")`` to the same flag
``annotate/order.py:_annotate_search_scores`` only emits the GROUP BY
when fan-out actually exists. Cover path stays gated by ``for_cover``.
5.4 — Skipped intentionally
``codex/views/auth.py`` already caches the three scalar inputs to
``get_acl_filter`` per request. The remaining cost is composing two Q
objects from those scalars — microseconds, not queries. Wrapping a
cached_property keyed on ``(model, user_id)`` would be cosmetic.
20 standalone cases of ``comic_filter_uses_m2m`` pass (every default-target
group, every m2m-folder TARGET, every BROWSER_FILTER_KEY by category, mixed
filters, ``pks=(0,)`` early-out).
stage5b-after.json: existing perf flows preserved (query counts identical;
wall times within run-to-run noise). The wins are on cold paths the harness
doesn't exercise — browsing a Series's comics, default-TARGET folder browse,
OPDS feeds — which a follow-up should add.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5c — batched choices_available (#588)
Replaces the per-field probe loop in BrowserChoicesAvailableView with a
single batched EXISTS annotate. FK fields keep "any non-null exists"
semantics; m2m fields decompose into (has_rel, has_null) booleans plus a
lazy distinct-count probe for the rare has_rel ∧ ¬has_null corner.
The natural EXISTS(SELECT DISTINCT rel ... LIMIT 1 OFFSET 1) form is
broken on SQLite — EXISTS short-circuits on the first row from the
underlying join, before DISTINCT collapses or OFFSET skips — so the m2m
path uses two cheap booleans + a Python-side cap-at-2 distinct probe.
Perf: flow_f_choices_available cold drops 34 → 11 queries (−68%) and
121 → 53 ms (−56%) on the dev DB. Other flows are noise-level
unchanged. tests/perf/run_baseline.py picks up three new flows
(choices_available, m2m field, FK field) so the changed code path is
visible in the artifact.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* run frontend with bun
* fix syntax error in filter sub menu
* Browser views perf: Stage 5d — has_metadata boolean cast + series-browse perf flow (#589)
Cleanup bundle (5.7) shrinks against post-5c code:
- #26: switch ``has_metadata`` annotation from ``F("metadata_mtime")`` to
``ExpressionWrapper(Q(metadata_mtime__isnull=False), BooleanField())`` in
both ``codex/views/browser/annotate/card.py`` and
``codex/views/reader/books.py``. Matches the consumer serializer's
``BooleanField`` and trims the SELECT projection to one byte per row.
- Harness: add ``flow_a2_series_browse`` so the harness covers the
Comic-queryset / no-m2m-filter path that Stage 5b's distinct + group_by
skip wins on. Headline measurement: 6 cold / 4 warm queries, ~13 ms cold.
Items #18 (already absorbed in 5c), #25 (parent-aware reduction already in
place; trimming further would change ORDER BY semantics), #30 (already
coalesced inside ``BookmarkUpdateMixin.update_bookmarks``), and #31
(``zipstream-ng`` already streams via ``FileResponse``) were re-investigated
and skipped with reasons documented in §10 of ``05-replan.md``.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff (#590)
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: 99-summary status column + OPDS perf plan (#591)
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: backlog status column on 99-summary
Stage 5 final exit criterion. Adds a Status column to every tier table in
tasks/browser-views-perf/99-summary.md so the backlog reads as a closed
ledger rather than an open plan.
Markers:
- Tier 1 (4): all four landed (Stage 2, 3, 4, 5a)
- Tier 2 (6): all six landed (Stage 1 / 5b)
- Tier 3 (7): four landed (Stage 2, 5b, 5c), one skipped (#16
subsumed by GroupACLMixin per-request scalar caches), three open
(#11, #12 absorbed by Stage 4 hints; #15 not on a hot Flow A-H path)
- Tier 4 (14): seven landed (Stage 1 / 5c / 5d), four re-investigated
in Stage 5d and confirmed already done (#25, #27, #30, #31), three
open (#22, #24, #29)
- Tier 5 (3): R1 ✅ Stage 5.9 (FTS demote regression test), R2 ❌
skipped (current code is correct), R3 ⏭️ deferred via the
stage5e-handoff-serializer-audit.md brief
Pure documentation update — no code or test changes. Closes the last
unchecked exit criterion on the Stage 5 plan.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: initial planning pass
Mirrors the structure of tasks/browser-views-perf/ for the OPDS surface.
No code changes — produces a ranked backlog so the same per-stage rigor
can be applied to OPDS once browser-views perf wraps.
Files:
- 00-meta-plan.md methodology, scope, sub-plan layout, exit criteria
- 01-routes-and-cache.md OPDS_TIMEOUT=0 disables cache_page on every feed
- 02-feed-pipeline.md v1 + v2 main feeds, BrowserView reuse, preview reruns
- 03-entry-serialization-v1.md v1 entries, lazy_metadata Comicbox open, M2M fan-out
- 04-publications-v2.md is_allowed static-method bypass of admin_flags cache
- 05-manifest.md credit fan-out 11→1, story_arcs N+1, subjects 7→1
- 06-progression-binary-aux.md PUT conflict pre-check + dead expr; binary inheritance
- 99-summary.md 5-tier ranked backlog, phasing A–F, cross-cutting guidance
Top findings (in landing order):
1. OPDS_TIMEOUT = 0 in codex/urls/const.py — every feed wraps
cache_page(OPDS_TIMEOUT) and gets nothing. Single-line config flip
is the highest-leverage win and gates Phase A.
2. Manifest credit fan-out (v2/manifest.py:194-199) — 11 separate
Credit.objects.filter queries because _MD_CREDIT_MAP is iterated.
Collapsible to one query + Python partition.
3. is_allowed static method (v2/feed/publications.py:35-56) bypasses
the request-cached admin_flags MappingProxyType; called per link
spec on start-page render.
4. Manifest M2M subjects — 7 queries via get_m2m_objects loop.
UNION or prefetch collapses.
5. get_publications_preview — full BrowserView pipeline rerun per
preview link spec on start page.
6. lazy_metadata() Comicbox open in v1 stream-link path —
synchronous file I/O on the request thread for partially-imported
books.
7. Progression PUT conflict pre-check — two queries per PUT,
foldable into one conditional UPDATE.
8. Story arcs N+1 in manifest — .only("story_arc", "number") defers
FK; per-row StoryArc.objects.get. Replace with select_related or
.values().
Plan only. No source files touched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* ignore tasks for prettier
* OPDS views perf: Stage 0 — baseline harness + Phase B low-risk wins (#592)
Closes Phase A (R1 OPDS_TIMEOUT=0 rationale, R2 perf harness) and four
of five Phase B items from `tasks/opds-views-perf/99-summary.md`:
- #3 Fix story-arc N+1 in `_publication_belongs_to_story_arcs` —
swap `.only("story_arc", "number")` for
`.select_related("story_arc").only("number", "story_arc__name")`
so `story_arc.name` access doesn't fire one query per row.
- #6 Convert `OPDS2PublicationBaseView.is_allowed` from `@staticmethod`
to instance method reading `self.admin_flags.get("folder_view")`,
and the parallel `OPDS1FacetsView._facet_group` anti-pattern
(`AdminFlag.objects.get` inside a per-facet loop) — both now use
the request-cached MappingProxyType from `SearchFilterView`.
- #13 Extract `_obj_ts(obj)` helper for the
`floor(datetime.timestamp(obj.updated_at))` expression repeated
at six sites across `v2/feed/publications.py` and `v2/manifest.py`.
- #15 Remove dead expression `max(position - 1, 0)` (no assignment)
at `v2/progression.py:226`.
Phase B #12 (`_update_feed_modified` rescan) deferred — it overlaps
with the preview-pipeline pass in Phase D.
The harness lives at `tests/perf/run_opds_baseline.py` with eleven
flows mirroring `tests/perf/run_baseline.py` shape. Cold + warm
captures via django-silk; cold pass invalidates cachalot +
django_cache. Captures `baseline.json` (pre-edits) and
`stage0-after.json` (post-edits) alongside the harness.
See `tasks/opds-views-perf/stage0.md` for the full writeup
including the harness reproducibility note.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 1 — manifest credit + subject batching (#593)
Closes Phase C from `tasks/opds-views-perf/99-summary.md`. Two
manifest items land:
- Tier 1 #2 — `_publication_credits` collapses 11 per-role
`Credit.objects.filter` calls into a single query with
`select_related("person", "role")`. Eliminates the 11-query loop
AND the lazy `credit.person` FK fan-out triggered by
`_add_tag_link` (7 queries on the dev DB's busiest comic). The
role-set partition runs in Python.
- Tier 2 #5 — `_publication_subject` collapses 7 per-model M2M
queries into a single `UNION ALL` over `(pk, name, _kind)` tuples.
Reconstructs `SimpleNamespace` rows so `_add_tag_link` and the
downstream `OPDS2SubjectSerializer` continue to work.
Drive-by: drop the now-unused `get_credits` helper from
`codex/views/opds/metadata.py` (no remaining callers).
`v2_manifest`: 47 → 24 cold queries (-23, ~49%), 113 → 87 ms cold.
Captured `stage1-before.json` + `stage1-after.json` alongside
`stage1.md` for the writeup including the surfaced (but not fixed
here) `peniciller` typo in `OPDS2PublicationMetadataSerializer`.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* speling
* OPDS views perf: Stage 2 — re-enable route caching (#595)
Closes Tier 1 #1 from `tasks/opds-views-perf/99-summary.md` — the
highest-impact item in the entire plan.
- `OPDS_TIMEOUT` flips from 0 (cache_page no-op) to 60 s. Long
enough to amortize a full feed pipeline run across a tab refresh
/ reader-app re-fetch, short enough that bookmark-position
changes show up before the next poll. The disable rationale is
reconstructed in stage0.md § R1.
- New `codex/urls/opds/__init__.py:opds_cached` helper composes
`cache_page(OPDS_TIMEOUT)` with
`vary_on_headers("Cookie", "Authorization")` so the cache key
scopes per-user / per-auth-scheme. Mirrors the binary cover-route
shape at `codex/urls/opds/binary.py`. Applied uniformly across
v1.py, v2.py, and the no-trailing-slash `/opds/v2.0` start in
root.py (which previously bypassed `cache_page` entirely).
- Progression route (`v2/<group>/<pk>/position`) explicitly NOT
wrapped — a PUT mutates the bookmark, and a GET within the cache
window would return stale position. Multi-device sync is the
worst-case (device A PUTs page 100, device B GETs within 60 s
and resumes at the wrong page). The ~9-query / 14 ms cold cost
is small compared to the freshness cost.
Warm-pass measurements collapse to 0 queries / ~1.5–2.7 ms across
every cacheable route (the cache returns the response without
entering the view layer):
v2_manifest 62 → 2.4 ms warm
v2_start 59 → 1.8 ms warm
v2_root_browse 28 → 2.4 ms warm
v1_root_browse 40 → 1.7 ms warm
v1_series_acquisition 24 → 2.7 ms warm
Cold-pass numbers unchanged — the view runs the full pipeline on a
cache miss. Real-world OPDS traffic is dominated by warm hits.
Cross-user isolation verified manually: User A and User B (different
ACL) hit the same URL and receive different payloads (16 223 B vs
15 217 B); each user's warm response matches their own cold
response; `Vary: Accept, Cookie, Authorization, origin` confirmed
on every response.
Captured `stage2-before.json` + `stage2-after.json` alongside
`stage2.md` for the writeup.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* disable discord notification steps in gha
* OPDS views perf: Stage 3 — preview pipeline cache sharing + book queryset joins (#596)
Closes Tier 1 #4 (preview-pipeline re-runs) and the related #17 /
sub-plan 02 #3 (`select_related` shortfall on the OPDS book
queryset).
- `OPDS2FeedLinksView.get_book_qs` overrides the parent
`BrowserView.get_book_qs` to add `select_related("volume",
"language")`. The base method joins `series` only, with an
explicit comment that "OPDS doesn't need volume" — but
`Comic.get_title(volume=True)` reads `obj.volume.name` /
`obj.volume.number_to` per publication, and `_publication_metadata`
reads `obj.language.name`. Without these joins, every publication
iteration fired one lazy `Volume.objects.get` and one lazy
`Language.objects.get`. v1 already does the same join in
`v1/facets.py:64`; this brings v2 to parity.
- `_get_publications_preview_feed_view` shares `_admin_flags` and
`_cached_visible_library_pks` with the parent view. Both are
request-scoped (depend on user, not on params/kwargs), so it's
safe to skip the per-preview re-fetch. Cuts the visible-library
ACL lookup from 1-per-preview to 1-per-request.
`v2_start`: 53 → 29 cold queries (-24, ~45% reduction). Pre-fix
breakdown: 15 codex_volume (N+1) + 12 codex_library (4 per preview
× 3) dominated. Post-fix: 0 codex_volume + 3 codex_library.
Out-of-scope hotspots still visible in the trace (documented in
stage3.md): per-preview age-rating-with-metadata join (4 queries,
needs the bigger UNION-batch rewrite) and the 9 codex_comic filter
/ annotation queries (preserved as the legitimate per-preview
pipeline cost).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 4 — v1 acquisition M2M batching + progression PUT conflict detection (#597)
Closes Phase E from `tasks/opds-views-perf/99-summary.md`.
#7 — Per-page batching of v1 entry M2M fan-out
==============================================
Three new helpers in `codex/views/opds/metadata.py`:
- `get_credit_people_by_comic` — one query for all credits across
all comics on the page; partitions by comic_id in Python.
- `get_m2m_objects_by_comic` — UNION ALL across the 7
`OPDS_M2M_MODELS` tables, partitioned by `(comic_id, kind)`.
`OPDS1EntryData` gains `authors_by_pk` / `contributors_by_pk` /
`category_groups_by_pk` optional dicts. `OPDS1FeedView._get_entries_section`
populates them when `metadata=True` and `key=="books"`. Per-entry
properties read from the dicts when present, fall back to the legacy
single-comic helpers otherwise (so facet entries / single-comic feeds
still work).
Result: 9 queries per entry × N entries collapses to 3 queries per
page. On the harness's "All Batman" series with `?opdsMetadata=1`
(106 comics), `v1_acquisition_with_metadata` drops from 817 cold
queries / 1585 ms to **20 cold queries / 154 ms** (~40× / ~10×) —
verified in a controlled full-feed-state run.
#8 — Progression PUT atomic conditional UPDATE
==============================================
Two changes:
1. `OPDS2ProgressionSerializer.modified` flips from
`read_only=True` to `required=False`. Previously the field was
silently dropped from PUT validated_data, making the conflict
pre-check at `view.py:207-217` unreachable (zero
progression-related queries fired on PUT today).
2. `OPDS2ProgressionView.put` replaces the dead pre-check (which
was `_get_bookmark_query() + qs.first()` — never executed) with
a single atomic conditional UPDATE keyed on
`updated_at__lte=new_modified`. If the UPDATE matches a row,
write succeeds in one query. If 0 rows match AND a bookmark
exists, the DB has a fresher row → 409. If 0 rows match AND no
bookmark exists, fall through to the existing async
`update_bookmark` path (first-time write).
Behavior change: clients that previously sent stale `modified` and
got silent 200s now correctly receive 409 per the OPDS v2
progression spec. Functional verification:
- PUT no `modified` (no bookmark) → 200 (liberal accept, async create)
- PUT with stale `modified` → 409 (atomic conflict detection)
- PUT with fresh `modified` → 200 (atomic UPDATE)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* bump news for progression bugfix
* OPDS views perf: Stage 5 — Tier 3-4 cleanups (#598)
Closes Phase F from `tasks/opds-views-perf/99-summary.md`. Three
items land; two are intentionally skipped after audit.
#16 — `select_related("parent_folder")` on manifest queryset
`OPDS2ManifestView.get_object` adds the join; eliminates the
lazy `Folder.objects.get` per request when folder_view is on.
v2_manifest cold drops 23 → 22 queries.
#11 — Memoize filters JSON via `self.params["filters"]`
`_subtitle_filters` previously re-parsed `request.GET["filters"]`
inline (urllib.parse.unquote + json.loads). The same JSON is
already parsed by BrowserSettingsFilterInputSerializer; reading
from `self.params` skips the third parse. Sub-ms per request,
but on a hot client refresh cumulative.
#10 — Resolve `opds:bin:page` URL once for `_publication_reading_order`
Replace per-page `self.href()` (which fires `reverse()` each call)
with a single sentinel-page resolution + `str.format` substitution.
Saves N-1 `reverse()` calls per manifest hit. Invisible on the
harness's 1-page comic_pk=10785; visible on high-page-count PDFs.
#12 (audited won't-fix): the rescan is functionally necessary —
preview-group mtimes aren't covered by `_get_group_and_books`'s
mtime; removing the rescan would lose preview mtime tracking.
#18 (audited won't-fix): Stage 4 already provides the cleaner
SimpleNamespace pattern; legacy `_add_url_to_obj` fires only on
cold fallback paths where the cachalot reuse risk doesn't apply
(materialized list, not a queryset reused for caching).
After Stage 5 the OPDS perf project has reached the point where
remaining open items either need production telemetry (R3, #19),
medium-risk correctness verification (#9), or are negligible wins
versus framework cost (#14). Recommending pause until production
data points to a specific path.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Reader views perf plan: methodology + 3 sub-plans + ranked backlog (#599)
Audit of `codex/views/reader/` (~850 LOC across 6 source files)
mirroring the OPDS / browser-views perf plan structure. Identifies
15 ranked items + 5 research questions across three view families:
- Reader view chain (`c/<pk>` GET) — params/arcs/books/reader.py
- Reader settings (`c/settings`, `c/<pk>/settings`) — settings.py
- Reader page binary (`c/<pk>/<page>/page.jpg`) — page.py
Top three findings:
1. Comicbox archive open on every page request (sub-plan 03 #1) —
200-page comic = 200 archive opens per read-thro…
ajslater
added a commit
that referenced
this pull request
May 4, 2026
* update deps
* v1.10.1 fix opds v2 links
* fix news
* fix gha trigger gate logic
* fix gha yaml bug
* fix gha build and deploy from firing on develop
* fix admin update user view & serializer to allow user updates.
* fix gha yaml syntax error'
* fix runing build on develop branch
* fix gha branch gate logic again
* skip tests on deploy if identical files. change discord colors & messages. move ci contaiiner to compose
* fix polling all libraries when no ids submitted in task
* fix poll all library button
* fix opds v2 manifest series link
* force browser reset on start page of opds v2
* fix dev csp for vite hmr
* working server folder picker with enter activates menu and waits to render so the menu isn't lost in the center of the screen
* clean up server folder code to be smaller and remove cruft
* bump news
* attempt to make granian auto reload not hold on to file handles by adjusting template settings in debug mode
* Squashed commit of the following:
commit c76660006840abb36aa37d7355d5c7e242babebf
Merge: b2b5a011a be94a0ae0
Author: AJ Slater <aj@slater.net>
Date: Sun Apr 5 15:49:11 2026 -0700
Merge branch 'develop' into select-multiple
commit b2b5a011ab207a7307505b092877f789e1a66ab3
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:15:54 2026 -0700
fix card menu hover highlighting
commit 5a98fe23d89be5e11fa46ada927100ccd3e08cda
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:13:37 2026 -0700
new design for select many mode. no settings drawer involvement. reconfigure cards
commit 82c612e3f77eec90b02465174b49fc8552871686
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:50:11 2026 -0700
add github config to source include. remove dockerhub config
commit b02bd450f4598300d78433a2d749741872d14894
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:42:32 2026 -0700
update deps
commit d7bbc875e7936d808259c9ba726127fcc3af905e
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 23:56:12 2026 -0700
select many toolbar
commit 246d5ecda59aaabb83e41db3ad32ec552804b9ae
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 22:23:24 2026 -0700
select many feature first attempt
* bump news
* efficiency and bugfixes for gha yaml
* fix container name for gha
* fix test report steps running if tests don't run
* use eslint_d for frontend lint
* make sure to build the ci compose locally
* fix too exuberant volumes mounting for gha compose
* remove working dir entry i'm not sure it's good
* add debug line to gha workflow
* remove debug line
* fix test make order
* stop docker compose when done in gha
* i don't think i need DJGNO_SETTINGS_MODULE th ci thing
* fix down call to own bring down ci task
* fix docker compose down"
* fix test results upload
* fix test step name
* fix gha permissions
* bump news
* update deps
* browser set_params method saves last route and saves params to settings
* bump version to 1.10.4
* modify gha discord notification
* use happy-dom for frontend tests
* docker tag script better than the one in ci/
* update deps
* far saner param initialization for browsers and opds start pages
* update dpes & bump alpha version
* v1.10.5a0 (#551)
* bump news
* cirlcle ci no longer handles pre-release
* remove alpha scripts for circleci"
* fix lint-ci
* fix pre-relase gha
* fix default last route on start pages
* bump version
* fix default params
* alpha2
* regular version 1.10.5
* remove develop circleci builds
* adjust variable name
* bump version to 6 alpha 0
* fix default params for feed_views in opds 2
* debug logging for django crash
* v1.10.6a0 (#553)
* bump news
* ignore ruff qa for debug build
* ignore shellcheck"
* lint
* minor refactors of opds v2 feed
* regular v1.10.6 version
* fix docker-tag-latest cscript
* ignore gh token file
* try to log more request errors
* reconfigure logging to hopefully be more verbose about request errors in production
* update deps and bump to alpha version
* bump news (#555)
* v1.10.7
* bump news
* update devenv
* update devevn and deps
* fix pm script
* move django-check to test category
* workflow build frontend and collect static for prodcution build. fix test upload
* bump version and news 1.10.8
* fix dev-module script
* fix news
* explain news
* fix opds clear search setting
* fix clear search button
* use a registry cache instead of gha cache for the dist-builder
* gha use more env vars for image names. retain python dist for 2 days.
* new quick deploy gha script. update deps & devenv.
* silence watchfiles 5 second timeout debug message
* consolidate null values const
* make scope private
* update devenv
* update deps. migrate to unhead v3
* update deps
* fix creating reader global settings
* fix caching
* rename codex build-dist to codex-ci
* fix image name. make gha steps depend on each other more.
* fix gha syntax errors
* names for gha steps
* use ghcr.io for python-debian base
* update deps
* format dockerfile
* picopt treestamps
* fix custom covers not importing. v1.10.11
* fix custom covers count in admin view
* bump news for custom cover count fix
* update deps
* codex identification in server tag and opds generator tag
* update deps
* force no entries on opds start page
* common opds start page mixin. emtpy group objects on start page
* update deps. typechecking.
* api change q to search
* standardize search param as 'search' instead of 'q' or other variations
* remove errant icecream
* clear settings on backend
* Squashed commit of the following:
Fix clear settings null bug, add global settings clear button
- clearComicSettings was setting book.settings to null, causing
Object.entries() to throw TypeError downstream in getBookSettings
- Add null guard in getBookSettings as defense-in-depth
- Add null guard in isClearDisabled computed
- Add clearGlobalSettings action and clear button to Default Settings panel
- Compare against READER_DEFAULTS to determine if global clear is disabled
* simplify settings class hierarchy
* rename select-many store to browser-select-many
* switch to bun. updated devenv
* add a claude md
* use frozenattrdict to speed up configuration
* fix rename of browserSelectMany store
* auth token help
* fix sort-ignores to make deterministic across shells with different locales
* fix crash on settings not being raw
* another gaurd for getMetadta()
* remove keys from unhead meta headers
* fix unhead description for admin tabs"
* fix overzealous lazy importer
* fix lazyImportEnabled variable in metadata-activator
* fix metadata activator from bad cherry pick
* fix errant quote
* fix typechecking
* update devenv & deps
* bump news
* fix import bug linking folders
* fix possible batching crashes. adjust import variables for throughput
* batch comic updates
* move INTERNAL_IPS setting to general django area
* fix typechecking issue
* update deps
* bump version to v1.10.12
* fix redirect on OPDS alternate view with metadata
* minor change to browser empty page for better first time experience
* allow browsing to comics with any top group in opds
* bring project up to speed with bun and no package-lock.json
* fix browser paginator
* bump news for browser paginaor fix
* fix search combobox clearing
* uppercase book close button
* version v1.10.13. update deps
* fix pdfs not displaying
* fix csp for pdfs
* fix OPDS FK constraint failure when session row is missing (#607)
iOS Panels (and other Basic-Auth OPDS clients) intermittently hit
sqlite3.IntegrityError: FOREIGN KEY constraint failed when settings
or bookmarks were saved. Two interacting bugs caused it:
- Janitor cleanup_sessions used `if not session.get_decoded():` to
detect "corrupt" sessions. get_decoded() returns {} for both real
decode failures and legitimate anonymous sessions with no stored
data — exactly what Basic-Auth OPDS clients produce. The nightly
task was wiping valid session rows. Replaced with a direct
signing.loads() call so only genuine signature/decode failures are
flagged.
- _ensure_session_key returned the cookie's session_key without
verifying the row still exists. With cached_db the session loads
from cache without rechecking, so a stale cookie key would slip
through and cause an FK violation when used as SettingsBrowser /
SettingsReader.session_id. Now we verify existence and flush+save
to cycle the key when the row is gone.
Either fix alone closes the user-visible error; both together also
stop the underlying churn that created the bad state.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* extend session-key validation to bookmark + reader-settings paths (#608)
The same stale-session_key FK-violation pattern existed in two more
places that also write rows whose session FK can be stale:
- BookmarkAuthMixin.get_bookmark_auth_filter — feeds session_id into
Bookmark.objects.bulk_create / bulk_update.
- ReaderSettingsBaseView._get_bookmark_auth_filter — feeds session_id
into SettingsReader.objects.create.
Both used the old `if not session.session_key: save()` pattern that
trusts the cookie. Hoist the validated _ensure_session_key helper
from SettingsBaseView up to AuthMixin so every auth-aware view shares
one implementation, and switch both call sites to it. BookmarkAuthMixin
now extends AuthMixin to inherit the helper. BookmarkFilterMixin is
unchanged — it's read-only (filter Q only) and a missing session
correctly returns no rows.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* format
* bump news
* Stats: fix user_registered_count / auth_group_count always-zero bug (#611)
The /admin/stats endpoint's user_registered_count and
auth_group_count fields have been silently returning 0 since at
least Sep 2024. The Stats tab in the admin UI shows 0 registered
users even on installs with multiple accounts.
Root cause: _add_config tried to rename the per-model count keys
produced by _get_model_counts:
config["user_registered_count"] = config.pop("users_count", 0)
config["auth_group_count"] = config.pop("groups_count", 0)
But _get_model_counts builds keys via
``snakecase(model.__name__) + "_count"``. For Django's
``django.contrib.auth.models.User`` / ``Group`` that produces
``user_count`` and ``group_count`` (singular). The pop()s with the
plural names never matched, so the default ``0`` won every time —
and the actual ``user_count`` / ``group_count`` keys were left
orphaned in the dict, then dropped by the StatsConfigSerializer
which only declares ``user_registered_count`` /
``auth_group_count``.
Fix: pop the right source keys.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* defaults for dockerfile ARGs
* format'
* Frontend correctness: 10 bug fixes from sub-plan 01 (#647)
* Reader page: fix load-progress spinner that never appeared
Two stacked bugs in the same setTimeout:
1. Non-arrow callback lost ``this``. The function ran with the
timer's context, not the component's, so ``this.loaded`` and
the write below were both no-ops.
2. The write targeted ``this.loading``, which has never been a
data field on this component. The template binds the spinner
to ``showProgress`` (line 15: ``v-if="showProgress && !loaded"``).
So even if the arrow had been there from the start, the spinner
still wouldn't have rendered — both bugs had to land at once.
Net: ``LoadingPage`` has been dead code for slow image loads.
Switch to an arrow function and write ``showProgress`` instead
of ``loading``. Stash the timer ID so ``beforeUnmount`` can
clear it; a fast page swap mid-delay would otherwise fire the
write on a torn-down component.
Implements B1 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: fix arc-mtime fallback that itself 500'd
``loadMtimes`` builds an arcs list of ``{ group, pks }`` from
``this.arcs``; if the dict is empty the function previously
fell back to ``arcs.push({ r: "0" })``. The comment noted that
"No arcs is a 500 from the mtime api" — the fallback was added
to dodge that 500 — but the wrong-shape fallback also produced
a 500 because the API expects ``group``/``pks`` keys, not ``r``.
Use the canonical shape so the fallback actually works.
Implements B2 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* iOS PWA download: pass the object URL to revokeObjectURL, not the Blob
``URL.createObjectURL(blob)`` returns a ``blob:...`` URL string;
``URL.revokeObjectURL`` must receive that same string to free the
mapping. The previous code passed ``response.data`` (the Blob
itself), which silently no-op'd and leaked one object URL per
download.
On iOS PWAs this matters more than elsewhere because the leak
accumulates across the user's session and can't be reclaimed
short of reloading the app.
Implements B6 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser API: stop mutating caller's settings in getGroupDownloadURL
``getGroupDownloadURL`` did ``delete settings.show`` on the
caller's object before building the URL. Side-effect: any caller
that re-used the settings dict after the download-URL build saw
its ``show`` key silently vanish. This was probably fine when
the function was first written but it's a footgun now that
settings flow through a Pinia store.
Destructure-and-spread to drop ``show`` without touching the
input.
Implements B8 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Auth store: make logout awaitable + clear state unconditionally
Two changes to the logout action:
1. ``async`` so callers can ``await``. The current call site
(``auth-menu.vue``) fires and forgets, but a future UX pass
that wants to disable the button while logout is in flight
needs the promise.
2. Clear ``this.user`` in ``finally`` rather than only on
success. The user clicked "log out" — UI should reflect the
logged-out state immediately, regardless of whether the
server-side logout endpoint succeeded. Server-side cookies
that survive the network failure will get cleaned up by the
next 401.
Implements B7 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser filter menu: stop double-rendering each filter row
``<v-list>`` was passed both ``:items="vuetifyItems"`` AND a
default-slot ``v-for`` over the same list. Vuetify renders the
items prop into ``v-list-item`` children directly, so every row
was being built twice — once by the prop, once by the manual
``v-for``. Visible to users on filter menus with large choice
lists (genres, characters, etc.); each row appeared duplicated
and the DOM cost doubled.
Drop the prop. Keep the ``v-for`` because it carries the custom
``#append`` slot for ``metronName`` rendering. ``:model-value`` /
``@update:selected`` still drive selection state via each list-
item's ``:value`` prop.
Implements B10 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Admin job-tab: remove API-fetching click handler from expanded panel
The expanded status panel had ``@click="loadAllStatuses"`` on its
container div. Any click inside the panel — including clicks on
child elements that bubbled — refetched the entire status map.
Probably copy-pasted as a "refresh on click" gesture, but it
fired far too often: a user inspecting a long status list would
trigger N API calls just from glancing around.
The status data is pushed through the websocket already
(socket.js fans librarian notifications into the admin store),
so the panel is up-to-date without a manual refresh.
Implements B11 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: handle bookmark-write errors instead of silently rejecting
``setRoutesAndBookmarkPage`` awaited ``_setBookmarkPage`` but
didn't catch its errors. On a network blip the promise rejected,
the bookmark didn't persist, and the failure became an unhandled
rejection in the browser console — not visible to the user, not
retried, just lost.
Wrap in try/catch. The local page state stays where it is (the
user is reading forward; the bookmark catches up on the next
write), but the failure is logged so debugging surfaces. A
proper user-visible toast + retry path is broader UX work
tracked in the plan.
Implements B3 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: clear progress timer on unmount
``updateProgress`` chains itself via ``setTimeout`` until the
metadata loads or progress reaches 100. The timer ID was never
stashed, so closing the dialog mid-animation left the chain
running — each tick fired on a torn-down component, writing
``this.progress`` and re-scheduling against now-null refs.
Stash the timer ID and clear it in ``beforeUnmount`` so the
chain stops cleanly when the dialog goes away.
Implements B12 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader pager: key dynamic component on identity to force remount
``<component :is="...">`` without ``:key`` lets Vue reuse the
existing instance across an ``is`` change when the components
share enough surface (props, name). For the reader's
vertical/horizontal pager swap that's wrong: scroll listeners
attached by the previous mode persist, the new mode's
``mounted`` runs against stale internal state, and any
abort/teardown logic in ``beforeUnmount`` never fires.
Add ``:key="component.name"`` so the swap is a true unmount +
remount — old listeners go away, new mode starts clean.
Implements B13 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: lint cleanup for the B12 fixup
Vue option-order rule: ``beforeUnmount`` belongs above
``methods``. Block-comment style required for the multi-line
explanation in ``updateProgress``. Both surfaced when running
eslint on the prior commit; pure cleanup, no behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* news for cherry pick
* OPDS auth: send WWW-Authenticate + opds-authentication content-type on 401 (#652)
Panels and other strict OPDS clients require the WWW-Authenticate header
(per RFC 7235) to trigger the auth prompt, and the spec calls for
application/opds-authentication+json on the auth document. The exception
handler was already converting 403 to 401 with the auth doc body, but
was bypassing DRF's natural 401 response and dropping both the header
and the proper content type, which Panels read as a forbidden state.
Also fix a stray `from re import DEBUG` in the auth view that always
forced the absolute-URL path.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix typing inheritence with drf perms
* bump version to v1.10.14
* update picopt treestamps
* fix typing import error
* fix vulture ignorelist
* readd the mistakenly removed vulture_ignorelist
* v1.10.15 fix show order bug. update deps
* fix nightly job manual expansion
* fix news version number
* V1.11 performance (#714)
* title for age rating tagged
* rename unrestricted to adult for clarity
* remove circleci code
* update to new devenv scripts in frontend
* fix sort-ignores to make deterministic across shells with different locales
* batch comic updates
* fix possible batching crashes. adjust import variables for throughput
* bump news for v1.11.0
* Add features to readme
* add saved view feature
* Add browser-views perf measurement harness (Stage 0) (#574)
Stand up the measurement infrastructure that gates the rest of the
browser-views performance work:
- Add django-silk 5.5 to the dev group; install + route it to a
separate ``silky`` SQLite DB via a new ``SilkRouter`` so profiler
traces never touch the live DB.
- Wire SilkyMiddleware below ServeStaticMiddleware so it only wraps
the API stack, not static-file responses.
- Expose /silk/ under the existing DEBUG URL block.
- Add ``tests/perf/run_baseline.py``: hits the live dev DB via
``django.test.Client``, runs three cold/warm flow pairs (root
browse, filtered search, series metadata), and writes a JSON
baseline artifact. Cachalot + page-cache are invalidated before
each cold pass so the numbers reflect actual DB work.
- Add ``make perf-baseline`` target.
- Commit the per-view analysis docs and initial baseline capture
under ``tasks/browser-views-perf/`` for reference during the
follow-on cleanup stages.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 1 - startup + annotation + filter micros (#575)
Bundle of small, surgical wins observed on the slimlib baseline:
- Cache `libraries_exist` with a 60s TTL in Django's default cache;
invalidate on Library save/delete via lazy-imported signal handlers.
Drops a redundant `Library.objects.filter(...).exists()` from every
browser response.
- Short-circuit the four `_save_browser_*` code paths in
`codex/views/settings.py` when nothing actually changed. Avoids
gratuitous `.save()` calls that flush cachalot on no-op PATCH writes.
- Dedupe the `add_group_by(qs)` call in
`codex/views/browser/browser.py`: group once in
`_get_common_queryset` (no-op for Comic), drop the second call in
`_get_group_queryset`.
- Memoize `get_max_bookmark_updated_at_aggregate` per
`(model, agg_func, default)` on the view instance — the three
callers (group_mtime, order, bookmark) now share one Aggregate.
- Move the `bmua_is_max` flag off the per-row `Value` annotation and
read it from `self.context["view"]` in the browser serializer.
- `@lru_cache(maxsize=256)` on `_preparse_search_query`: extract to a
pure module-level helper keyed on `(text, path_allowed)`; returns
frozen tokens.
- `@lru_cache(maxsize=512)` on `get_field_query`: cache the Lark-parsed
Q tree, `copy.deepcopy` on return because `_hoist_filters` mutates
`child.negated` downstream.
- Stash the `BaseDatabaseOperations(None)` singleton as `_DB_OPS` in
`search/field/expression.py`; `prep_for_like_query` doesn't use the
connection.
- Pre-filter the field loop in `filters/field.py` to keys actually set
in the request — saves ~20 no-op calls per browser.
- Delete `search/field/optimize.py` (`like_qs_to_regex_q` and friends
were already unused — grep confirms only self-references).
Slimlib cold baseline (3 flows, stage1.json vs Stage 0 baseline.json):
- flow_a_root_browse: 21→18 SQL, 189.6→182.3 ms
- flow_b_filtered_search: 21→17 SQL, 187.4→178.0 ms
- flow_c_series_metadata: 34→31 SQL, 251.1→226.9 ms
Warm paths unchanged (0 SQL, ~2 ms). Full tests + ruff pass.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix type warnings
* fix lint error
* format
* Run silk migrations on silky DB at startup (#576)
Stage 0 wired a second SQLite DB (silky) for django-silk profiling traces
and a DATABASE_ROUTERS entry that routes silk app models there. But
ensure_db_schema only invoked `call_command("migrate")` without a database
arg, which only migrates the default DB. The router then blocks silk
migrations from running on default — so the silky DB was never populated,
and the first request through SilkyMiddleware failed with
"no such table: silk_request".
Mirror the pattern from tests/perf/run_baseline.py: after the default
migrate, also run `migrate silk --database=silky` when the silky DB is
configured. Guarded on `"silky" in connections.databases` so production
(DEBUG=False, no silky DB) is a no-op.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 2 — triple COUNT + page mtime cache (#577)
## PR 2a — Eliminate triple COUNT on the paginate path
Each browse request ran three COUNT queries per section (groups & books):
1. The outer grouped COUNT in `_get_common_queryset`.
2. Paginator's internal COUNT triggered by `paginator.page()`.
3. An explicit `.count()` on the paginated slice.
The outer COUNT is needed (sizing the paginator). The other two are
redundant — the page row count is bounded by `per_page` and derivable
from `end_index - start_index + 1`.
- Shadow `Paginator.count` (a `@cached_property`) with the pre-computed
total to skip Paginator's internal COUNT.
- Derive page row count arithmetically from `Page.start_index()` /
`end_index()`.
- Pass `book_count` through `paginate()` alongside `group_count`; drop
`book_qs.count()` on the opds2 path.
- `_paginate_section` returns `(qs, count)` directly.
Short-circuits on `total_count == 0` (avoids Paginator instantiation on
empty sections) and preserves the EmptyPage warning branch.
## PR 2b — Short-TTL page mtime cache
`BrowserView._get_page_mtime()` calls `get_group_mtime(page_mtime=True)`
on every browse request. The query is a filtered Max aggregate that
cachalot caches — but any write to Comic / Bookmark invalidates it, so
bookmark-heavy usage forces recomputation. Cold-path silk traces show
this aggregate at ~26ms on flow_a — the second-slowest query in the
request.
Add a 5s TTL cache layer gated on page_mtime=True. Key includes user,
model, group, pks, page, and a hash of filter-affecting params
(filters, search, q, order_by, order_reverse). The polling MtimeView
path (no page_mtime) is unaffected, so frontend change-detection stays
live.
## Measurements
tests/perf/run_baseline.py on the slimlib DB. Cold = full cache
invalidation; warm = cachalot populated.
| flow | stage1 cold | stage2 cold |
|-------------------------|------------------------|------------------------|
| flow_a root browse | 18 queries / 182.3 ms | 16 queries / 135.6 ms |
| flow_b filtered search | 17 queries / 178.0 ms | 15 queries / 130.3 ms |
| flow_c series metadata | 31 queries / 226.9 ms | 31 queries / 229.9 ms |
flow_a / flow_b: -2 queries, ~26% cold wall-time reduction. flow_c
unaffected (metadata doesn't traverse paginate). PR 2b's benefit is a
dogpile guard after cachalot invalidation — doesn't show in the harness
(cold = both caches empty; warm = cachalot wins first).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix browser paginator
* typecheck browser paginate
* Browser views perf: Stage 3 — cover fan-out collapse (#578)
* Stage 3 — cover fan-out: per-pk endpoint + pre-resolved cover_pk
Before stage 3, every cover card triggered a full BrowserAnnotateOrderView
pipeline to pick the representative comic pk. A default page does ~100 of
these in parallel. This PR collapses that into one correlated Subquery on
the browse response plus a thin per-pk cover endpoint.
Components
- Option A: annotate_order_aggregates(for_cover=True) drops the JsonGroupArray
+ page_count aggregates from the cover path — unused for picking a pk.
- Option B.1: BrowserView group cards get cover_pk / cover_custom_pk via
correlated Subquery that replicates CoverView.get_group_filter exactly
(direct fk match when dynamic_covers/Volume/Folder; sort_name fuzzy match
otherwise, correlated on _GROUP_BY columns so the same comic set as `ids`
is picked — no JSON parsing, no peer aggregate).
- New endpoints /api/v3/c/<pk>/cover.webp and /api/v3/cc/<pk>/cover.webp
serve already-resolved pks with a cheap single-row ACL probe and the
existing CoverPathMixin / CoverCreateThread pipeline.
- Frontend getCoverSrc prefers the new per-pk URL when cover_pk /
cover_custom_pk is on the card; falls back to the old group+pks URL
otherwise, so OPDS and search-active browses keep working.
- FTS skip: cover_pk annotation is skipped when params["search"] is set.
MATCH inside a correlated subquery re-scans the FTS5 index per outer
row (~900ms on a 100-group page). The old URL path still applies the
search filter per cover — same behavior, parallelized over HTTP.
Perf (slimlib dev DB, Flow D = browse + every card's cover):
Flow Before cold After cold Delta
A root browse 156.9ms / 15 q 180.5ms / 15 q +24ms / +0 q
B search 148.3ms / 16 q 132.4ms / 16 q -16ms / +0 q
C metadata 161.9ms / 31 q 165.0ms / 31 q +3ms / +0 q
D browse+covers 2311ms / 1216q 1161ms / 815 q -50% / -33%
Flow A takes a mild regression for the correlated cover subquery, but Flow
D — the realistic user wall-clock — drops by half and 400 SQL queries.
Subsequent cover fetches drop from ~90ms each (full pipeline) to ~5ms
(disk read + 1 ACL probe).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Stage 3 follow-ups: custom_cover URL, FTS pre-materialization, OPDS thin covers
Address review feedback on the cover fan-out collapse:
- Rename /api/v3/cc/<pk>/cover.webp → /api/v3/custom_cover/<pk>/cover.webp.
Descriptive over terse; matches the view name.
- FTS pre-materialization replaces the FTS-skip fallback. A correlated
MATCH re-scans FTS5 per outer row (~900ms on a 100-group page); the
old path worked around this by skipping cover_pk annotation on search
and sending every cover through the legacy pipeline once. We now
pre-select the FTS match set as a non-correlated sub-SELECT in the
outer cover subquery — SQLite materializes it once and each correlated
cover row lookup becomes an indexed pk filter. Cover_pk is annotated
on search responses, the thin endpoint handles each cover.
- OPDS now emits thin-endpoint cover URLs (v1 and v2):
- New OPDSComicCoverByPkView / OPDSCustomCoverByPkView wrap the browser
thin views with OPDSAuthMixin's Basic Auth.
- New opds:bin:cover_by_pk and opds:bin:custom_cover_by_pk URL names.
- v1 _cover_link picks: cover_custom_pk → custom thin; group=='c' → own
pk thin; cover_pk → thin; else legacy group+pks fallback.
- v2 _thumb always uses the thin endpoint (publications are Comic rows
so obj.pk IS the cover pk).
- OPDS inherits annotate_cover() via BrowserView._get_group_and_books,
so group rows already carry cover_pk / cover_custom_pk.
- tests/perf/run_baseline.py gains a flow_e (search + covers) to measure
the search path end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix search combobox clearing
* format
* Stage 3 follow-ups: folder covers, endpoint rename, offline cover pipeline (#579)
- Fix folder cover_pk picking comics not actually in a folder's subtree
by using the recursive ``folders`` M2M (same relation the browse filter
uses) for the per-card cover subquery instead of the direct
``parent_folder`` FK.
- Delete the legacy thick cover endpoint. Rename cover_by_pk.py to
cover.py and drop the "_by_pk" suffix from view class names
(CoverView, CustomCoverView, OPDSCoverView, OPDSCustomCoverView).
Update URL configs, OPDS wrappers, and the frontend client.
- Move all cover generation off the HTTP path. Add CoverCreateTask and
enqueue it from the importer after bulk_create so new comics get
thumbnails pre-warmed offline. When a cached thumb is missing the
per-pk endpoint enqueues the task and responds 202 Accepted with
Retry-After plus the missing-cover placeholder instead of synthesizing
the WebP inline under a worker thread.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update devenv sort ignores
* Force cover refresh after 202 so the placeholder doesn't stick (#580)
Browsers don't honor Retry-After on <img> elements and happily cache the
placeholder served with the 202 response, so even after the cover thread
finishes writing the real thumb the img src keeps rendering the stale
placeholder bytes.
- Backend sends Cache-Control: no-store on the 202 placeholder so the
response isn't cached at that URL.
- BookCover now probes the cover URL with fetch() on mount and, if it
sees 202, waits Retry-After and bumps a reactive `retry` counter that
becomes a cache-busting query param on coverSrc. v-img re-fetches with
the new URL and gets the real cover once the cover thread is done.
Retries are capped at 5 and aborted on unmount via AbortController.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Harden cover endpoint against concurrent read/write races (#581)
Under a full cold-cache page load, 32 HTTP workers hitting the cover
endpoint at the same time as the cover thread was writing thumbs
returned 500s for a handful of covers. The endpoint used a three-step
exists()/stat()/read_bytes() sequence that could race against an
in-flight write, and save_cover_to_cache wrote directly to the target
path so readers could observe a truncated file mid-write.
- save_cover_to_cache now stages to a ``{name}.{pid}.tmp`` sibling and
renames with Path.replace, so a read either sees the pre-existing
state or the fully written file — never a partial one. Stale temps
are unlinked on failure.
- _get_cover_response collapses to a single read_bytes() that catches
FileNotFoundError and logs OSError, then falls through to the 202
path. No race window between stat and read.
- LIBRARIAN_QUEUE.put and the outer get() methods are wrapped in
try/except that log and return the placeholder, so a surprise
exception is never user-visible as a 500 and always leaves a log line.
- cleanup_orphan_covers runs a dedicated ``*.tmp`` sweep over both cover
roots before the orphan scan, so stale temps from crashed writers
don't accumulate.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix custom cover generation
* relegate watch for changes restart to a en env variable settings
* await and finish watch task
* update devenv
* Fix metadata cover fetching pk=0 after Stage 3 cover fan-out (#583)
Stage 3's follow-up commit (2ff08524) dropped the legacy `group+pks`
fallback from the frontend `getCoverSrc()` helper, leaving it to only
build per-pk URLs from `coverPk` / `coverCustomPk`. The browser card
pipeline annotates those fields via `BrowserAnnotateCoverView`, but the
metadata endpoint was never wired into that annotation, and
`metadata-cover.vue` never passed the cover pks through to `BookCover`.
As a result every metadata dialog tried to load
`/api/v3/c/0/cover.webp` — pk 0 — and fell back to the missing-cover
placeholder.
Wire the annotation + serializer + component end-to-end:
- `MetadataAnnotateView` inherits from `BrowserAnnotateCoverView` so
`annotate_cover()` is available on the metadata queryset.
- `MetadataView.get_object()` calls `annotate_cover(qs)` after
`annotate_card_aggregates(qs)`, mirroring `browser.py`'s ordering.
- `MetadataSerializer` exposes `cover_pk` and `cover_custom_pk` as
`SerializerMethodField`s with the same fallback-to-`obj.pk` semantics
that `BrowserCardSerializer` uses — so Comic metadata (`group=c`)
works without annotation, falling back to its own pk.
- `metadata-cover.vue` forwards `md.coverPk` / `md.coverCustomPk` to
`BookCover`.
Verified on /api/v3/{p,i,s,v,c,f}/*/metadata: `coverPk` now resolves
to a real comic pk on every group, and the comic path falls back to
its own pk. No SQL query delta — `annotate_cover` is a correlated
Subquery that inlines into the outer SELECT.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hints (#582)
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hydration hints
Fold the three `_intersection_annotate()` passes (value fields, conflict
shadow fields, related-count fields) into a single batched call that
collapses N distinct-count probes into one `aggregate()` and one
`values_list()`, with unique synthetic keys so annotation groups don't
collide.
Extend `FK_QUERY_OPTIMIZERS` so intersecting FK hydration
(`AgeRating`, `Character`, `Team`) carries `select_related` + `.only()`
hints for the nested fields the metadata serializer actually reads
(`AgeRating.metron`, `Character.identifier.url`). Without this, each
nested access fired a follow-up query per instance.
For the M2M intersection path, short-circuit empty-intersection fields
with `Model.objects.none()` so we skip the optimizer setup (and, for
the Comic self-reference path, pointless prefetch dispatches on already
empty results).
Add `flow_c2_comic_metadata` to the perf baseline harness so the
comic-detail metadata path (the FK + M2M heavy flow) is tracked
alongside the series flow.
Measured: flow_c_series_metadata cold SQL drops from 31 → 28 (-3).
flow_c2_comic_metadata baseline captured at 47 queries for future
stages. Other flows unchanged within noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* WATCH for changes variable depends on debug
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Fix folder + FTS crash and preserve rank-ordered cover selection (#584)
* Fix folder browse + FTS crash on search_score ordering
Browsing folders (or any group) with an FTS search and
``orderBy=search_score`` blew up with::
sqlite3.OperationalError: no such column: codex_comicfts.rank
Root cause: Stage 3's cover fan-out collapse builds a correlated
Comic subquery per group card to pick ``cover_pk``. The follow-up
(#579) replaced the correlated ``comicfts__match`` filter with a
non-correlated ``pk__in fts_sq`` pre-materialization — so the Comic
subquery no longer joins ``codex_comicfts``. But
``annotate_order_aggregates`` still annotated
``search_score=ComicFTSRank()`` inside the subquery, and
``add_order_by`` issued ``ORDER BY "codex_comicfts"."rank" * -1``,
which SQLite can't resolve from the subquery's FROM list.
- ``_annotate_search_scores`` now takes ``for_cover`` and skips the
annotation when set. ``annotate_order_aggregates`` threads the flag
through (matching the existing ``for_cover`` pipeline-trims).
- ``_cover_comic_subquery`` passes ``order_key="sort_name"`` to
``add_order_by`` whenever the user's order is ``search_score`` — the
cover's tie-break isn't user-visible, and ``sort_name`` is a real
indexed column on Comic.
Verified: /api/v3/{r,f,a,p}/0/1?q=iron+man&orderBy=search_score all
return 200 with a real ``coverPk`` (not 0). Lint + 20-test pytest
suite pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Preserve FTS rank ordering in cover subquery
The previous fix short-circuited to ``sort_name`` when ``order_by`` was
``search_score`` so a group card's cover could be picked without
referencing ``codex_comicfts.rank``. That fixed the crash but meant the
cover chosen for each card was the alphabetically-first matching comic,
not the top-ranked FTS match — a UX regression on searched pages.
Restore rank-ordered cover selection by:
* Applying ``fts_q`` (``comicfts__match=...``) directly in the cover
subquery so ``codex_comicfts`` is joined and ``rank`` is populated,
while keeping the ``pk__in`` pre-materialization for cheap filtering.
* Teaching ``ComicFTSRank`` to resolve the query-local alias for
``codex_comicfts`` at compile time. Its literal template worked for
the top-level browse query but emitted an unresolvable column ref in
nested subqueries where Django aliases the join as ``V4``/``U1``.
* Skipping ``.group_by("id")`` in the cover-subquery search_score
annotation. The custom force-group-by compiler emits a literal
``"codex_comic"."id"`` that breaks under nested aliasing, and the
cover subquery is already ``.distinct() … LIMIT 1`` so dedup is
redundant there.
Verified against the dev DB: for each publisher card on a ``batman``
search, the annotated ``coverPk`` now matches the top-ranked comic
returned by a direct ``ComicFTSRank`` ordering within that publisher.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update for claude rules
* Browser views perf: Stage 5a — server-side cache_page on cover endpoints (#585)
Wrap /api/v3/c/<pk>/cover.webp, /api/v3/custom_cover/<pk>/cover.webp,
and their OPDS counterparts in cache_page(COVER_MAX_AGE). Compose with
cache_control + vary_on_cookie (API) or vary_on_headers("Cookie",
"Authorization") (OPDS, which accepts Basic + Bearer + Session auth)
so the Vary header is set before cache_page stores the response — the
cache key is keyed per auth identity, no cross-user leakage.
Also raise Django FileBasedCache MAX_ENTRIES from the default 300 to
10000. Cachalot query results + cache_page entries (browser + cover)
exceed 300 during a single browse-with-covers pageload, triggering the
2/3 random cull that silently evicts just-populated cover entries
before the next request can read them. Without this, Flow D warm only
dropped to 743 (~50% cover-cache hit rate); with it, Flow D warm drops
to 0.
Perf impact (stage5a-after.json vs. stage4-after.json):
Flow D — browse + 100 covers warm 802 → 0 queries
Flow E — search + 46 covers warm 368 → 232 queries
Flow E's residual 232 queries are 29 covers that return 202 Accepted
(cover not yet generated; response has Cache-Control: no-store, which
cache_page correctly skips). The 17 covers that returned 200 were 0
queries each. In production, 202s resolve within seconds as the cover
thread catches up, so steady-state warm on Flow E is also 0.
Cold query counts and Flow A/B/C are unchanged.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Raise RLIMIT_NOFILE on startup to avoid macOS 256 FD cap (#586)
Cold browser sessions loading a full 100-card grid intermittently crash
the dev server with `OSError: [Errno 24] Too many open files`. Diagnosis:
- macOS ships a 256 soft cap for RLIMIT_NOFILE (`ulimit -Sn 256`).
- Each Django request thread keeps a sticky SQLite connection
(`CONN_MAX_AGE=600`).
- SQLite WAL mode opens 3 FDs per connection (main + `-wal` + `-shm`).
- The thin cover endpoint dispatches to a `sync_to_async` threadpool, so
a burst of 100 cold cover requests easily spawns ~100 threads → ~300
SQLite FDs, blowing past the 256 cap before page reads even open.
Bumping the soft limit toward the hard cap (or 8192, whichever is lower)
at process start is non-invasive and matches what production deployments
typically achieve via shell `ulimit`. No-op on Linux (already high) and
on platforms without the `resource` module.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps, format json
* Browser views perf: Stage 5b — annotation gating + m2m-aware distinct (#587)
Three surgical correctness/perf changes that surface the right SQL for the
right target. The Stage 5a work (cover endpoint cache_page) collapsed Flow D
warm to 0; 5b cleans up cold-path waste the harness flows don't directly
exercise.
5.2 — Skip ``updated_ats`` JsonGroupArray outside browser/metadata
``obj.updated_ats`` is consumed only in
``BrowserAggregateSerializerMixin.get_mtime`` (browser + metadata
serializers). OPDS computes its own mtime from the bookmark aggregate;
cover/download paths never read it. Skip the DISTINCT scalar aggregate
for those targets.
5.3 — m2m-aware .distinct() in ``BrowserFilterView.get_filtered_queryset``
New ``comic_filter_uses_m2m`` cached property. Comic queries skip
``.distinct()`` unless a real m2m or m2m-through join is present
(story_arc browse, folder browse on cover/choices/bookmark/download
TARGETs, or any m2m field filter). Non-Comic queries still always
``.distinct()`` because the ACL alone traverses ``comic__``
(one-to-many).
5.5 — Tie ``search_score`` ``group_by("id")`` to the same flag
``annotate/order.py:_annotate_search_scores`` only emits the GROUP BY
when fan-out actually exists. Cover path stays gated by ``for_cover``.
5.4 — Skipped intentionally
``codex/views/auth.py`` already caches the three scalar inputs to
``get_acl_filter`` per request. The remaining cost is composing two Q
objects from those scalars — microseconds, not queries. Wrapping a
cached_property keyed on ``(model, user_id)`` would be cosmetic.
20 standalone cases of ``comic_filter_uses_m2m`` pass (every default-target
group, every m2m-folder TARGET, every BROWSER_FILTER_KEY by category, mixed
filters, ``pks=(0,)`` early-out).
stage5b-after.json: existing perf flows preserved (query counts identical;
wall times within run-to-run noise). The wins are on cold paths the harness
doesn't exercise — browsing a Series's comics, default-TARGET folder browse,
OPDS feeds — which a follow-up should add.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5c — batched choices_available (#588)
Replaces the per-field probe loop in BrowserChoicesAvailableView with a
single batched EXISTS annotate. FK fields keep "any non-null exists"
semantics; m2m fields decompose into (has_rel, has_null) booleans plus a
lazy distinct-count probe for the rare has_rel ∧ ¬has_null corner.
The natural EXISTS(SELECT DISTINCT rel ... LIMIT 1 OFFSET 1) form is
broken on SQLite — EXISTS short-circuits on the first row from the
underlying join, before DISTINCT collapses or OFFSET skips — so the m2m
path uses two cheap booleans + a Python-side cap-at-2 distinct probe.
Perf: flow_f_choices_available cold drops 34 → 11 queries (−68%) and
121 → 53 ms (−56%) on the dev DB. Other flows are noise-level
unchanged. tests/perf/run_baseline.py picks up three new flows
(choices_available, m2m field, FK field) so the changed code path is
visible in the artifact.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* run frontend with bun
* fix syntax error in filter sub menu
* Browser views perf: Stage 5d — has_metadata boolean cast + series-browse perf flow (#589)
Cleanup bundle (5.7) shrinks against post-5c code:
- #26: switch ``has_metadata`` annotation from ``F("metadata_mtime")`` to
``ExpressionWrapper(Q(metadata_mtime__isnull=False), BooleanField())`` in
both ``codex/views/browser/annotate/card.py`` and
``codex/views/reader/books.py``. Matches the consumer serializer's
``BooleanField`` and trims the SELECT projection to one byte per row.
- Harness: add ``flow_a2_series_browse`` so the harness covers the
Comic-queryset / no-m2m-filter path that Stage 5b's distinct + group_by
skip wins on. Headline measurement: 6 cold / 4 warm queries, ~13 ms cold.
Items #18 (already absorbed in 5c), #25 (parent-aware reduction already in
place; trimming further would change ORDER BY semantics), #30 (already
coalesced inside ``BookmarkUpdateMixin.update_bookmarks``), and #31
(``zipstream-ng`` already streams via ``FileResponse``) were re-investigated
and skipped with reasons documented in §10 of ``05-replan.md``.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff (#590)
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: 99-summary status column + OPDS perf plan (#591)
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: backlog status column on 99-summary
Stage 5 final exit criterion. Adds a Status column to every tier table in
tasks/browser-views-perf/99-summary.md so the backlog reads as a closed
ledger rather than an open plan.
Markers:
- Tier 1 (4): all four landed (Stage 2, 3, 4, 5a)
- Tier 2 (6): all six landed (Stage 1 / 5b)
- Tier 3 (7): four landed (Stage 2, 5b, 5c), one skipped (#16
subsumed by GroupACLMixin per-request scalar caches), three open
(#11, #12 absorbed by Stage 4 hints; #15 not on a hot Flow A-H path)
- Tier 4 (14): seven landed (Stage 1 / 5c / 5d), four re-investigated
in Stage 5d and confirmed already done (#25, #27, #30, #31), three
open (#22, #24, #29)
- Tier 5 (3): R1 ✅ Stage 5.9 (FTS demote regression test), R2 ❌
skipped (current code is correct), R3 ⏭️ deferred via the
stage5e-handoff-serializer-audit.md brief
Pure documentation update — no code or test changes. Closes the last
unchecked exit criterion on the Stage 5 plan.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: initial planning pass
Mirrors the structure of tasks/browser-views-perf/ for the OPDS surface.
No code changes — produces a ranked backlog so the same per-stage rigor
can be applied to OPDS once browser-views perf wraps.
Files:
- 00-meta-plan.md methodology, scope, sub-plan layout, exit criteria
- 01-routes-and-cache.md OPDS_TIMEOUT=0 disables cache_page on every feed
- 02-feed-pipeline.md v1 + v2 main feeds, BrowserView reuse, preview reruns
- 03-entry-serialization-v1.md v1 entries, lazy_metadata Comicbox open, M2M fan-out
- 04-publications-v2.md is_allowed static-method bypass of admin_flags cache
- 05-manifest.md credit fan-out 11→1, story_arcs N+1, subjects 7→1
- 06-progression-binary-aux.md PUT conflict pre-check + dead expr; binary inheritance
- 99-summary.md 5-tier ranked backlog, phasing A–F, cross-cutting guidance
Top findings (in landing order):
1. OPDS_TIMEOUT = 0 in codex/urls/const.py — every feed wraps
cache_page(OPDS_TIMEOUT) and gets nothing. Single-line config flip
is the highest-leverage win and gates Phase A.
2. Manifest credit fan-out (v2/manifest.py:194-199) — 11 separate
Credit.objects.filter queries because _MD_CREDIT_MAP is iterated.
Collapsible to one query + Python partition.
3. is_allowed static method (v2/feed/publications.py:35-56) bypasses
the request-cached admin_flags MappingProxyType; called per link
spec on start-page render.
4. Manifest M2M subjects — 7 queries via get_m2m_objects loop.
UNION or prefetch collapses.
5. get_publications_preview — full BrowserView pipeline rerun per
preview link spec on start page.
6. lazy_metadata() Comicbox open in v1 stream-link path —
synchronous file I/O on the request thread for partially-imported
books.
7. Progression PUT conflict pre-check — two queries per PUT,
foldable into one conditional UPDATE.
8. Story arcs N+1 in manifest — .only("story_arc", "number") defers
FK; per-row StoryArc.objects.get. Replace with select_related or
.values().
Plan only. No source files touched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* ignore tasks for prettier
* OPDS views perf: Stage 0 — baseline harness + Phase B low-risk wins (#592)
Closes Phase A (R1 OPDS_TIMEOUT=0 rationale, R2 perf harness) and four
of five Phase B items from `tasks/opds-views-perf/99-summary.md`:
- #3 Fix story-arc N+1 in `_publication_belongs_to_story_arcs` —
swap `.only("story_arc", "number")` for
`.select_related("story_arc").only("number", "story_arc__name")`
so `story_arc.name` access doesn't fire one query per row.
- #6 Convert `OPDS2PublicationBaseView.is_allowed` from `@staticmethod`
to instance method reading `self.admin_flags.get("folder_view")`,
and the parallel `OPDS1FacetsView._facet_group` anti-pattern
(`AdminFlag.objects.get` inside a per-facet loop) — both now use
the request-cached MappingProxyType from `SearchFilterView`.
- #13 Extract `_obj_ts(obj)` helper for the
`floor(datetime.timestamp(obj.updated_at))` expression repeated
at six sites across `v2/feed/publications.py` and `v2/manifest.py`.
- #15 Remove dead expression `max(position - 1, 0)` (no assignment)
at `v2/progression.py:226`.
Phase B #12 (`_update_feed_modified` rescan) deferred — it overlaps
with the preview-pipeline pass in Phase D.
The harness lives at `tests/perf/run_opds_baseline.py` with eleven
flows mirroring `tests/perf/run_baseline.py` shape. Cold + warm
captures via django-silk; cold pass invalidates cachalot +
django_cache. Captures `baseline.json` (pre-edits) and
`stage0-after.json` (post-edits) alongside the harness.
See `tasks/opds-views-perf/stage0.md` for the full writeup
including the harness reproducibility note.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 1 — manifest credit + subject batching (#593)
Closes Phase C from `tasks/opds-views-perf/99-summary.md`. Two
manifest items land:
- Tier 1 #2 — `_publication_credits` collapses 11 per-role
`Credit.objects.filter` calls into a single query with
`select_related("person", "role")`. Eliminates the 11-query loop
AND the lazy `credit.person` FK fan-out triggered by
`_add_tag_link` (7 queries on the dev DB's busiest comic). The
role-set partition runs in Python.
- Tier 2 #5 — `_publication_subject` collapses 7 per-model M2M
queries into a single `UNION ALL` over `(pk, name, _kind)` tuples.
Reconstructs `SimpleNamespace` rows so `_add_tag_link` and the
downstream `OPDS2SubjectSerializer` continue to work.
Drive-by: drop the now-unused `get_credits` helper from
`codex/views/opds/metadata.py` (no remaining callers).
`v2_manifest`: 47 → 24 cold queries (-23, ~49%), 113 → 87 ms cold.
Captured `stage1-before.json` + `stage1-after.json` alongside
`stage1.md` for the writeup including the surfaced (but not fixed
here) `peniciller` typo in `OPDS2PublicationMetadataSerializer`.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* speling
* OPDS views perf: Stage 2 — re-enable route caching (#595)
Closes Tier 1 #1 from `tasks/opds-views-perf/99-summary.md` — the
highest-impact item in the entire plan.
- `OPDS_TIMEOUT` flips from 0 (cache_page no-op) to 60 s. Long
enough to amortize a full feed pipeline run across a tab refresh
/ reader-app re-fetch, short enough that bookmark-position
changes show up before the next poll. The disable rationale is
reconstructed in stage0.md § R1.
- New `codex/urls/opds/__init__.py:opds_cached` helper composes
`cache_page(OPDS_TIMEOUT)` with
`vary_on_headers("Cookie", "Authorization")` so the cache key
scopes per-user / per-auth-scheme. Mirrors the binary cover-route
shape at `codex/urls/opds/binary.py`. Applied uniformly across
v1.py, v2.py, and the no-trailing-slash `/opds/v2.0` start in
root.py (which previously bypassed `cache_page` entirely).
- Progression route (`v2/<group>/<pk>/position`) explicitly NOT
wrapped — a PUT mutates the bookmark, and a GET within the cache
window would return stale position. Multi-device sync is the
worst-case (device A PUTs page 100, device B GETs within 60 s
and resumes at the wrong page). The ~9-query / 14 ms cold cost
is small compared to the freshness cost.
Warm-pass measurements collapse to 0 queries / ~1.5–2.7 ms across
every cacheable route (the cache returns the response without
entering the view layer):
v2_manifest 62 → 2.4 ms warm
v2_start 59 → 1.8 ms warm
v2_root_browse 28 → 2.4 ms warm
v1_root_browse 40 → 1.7 ms warm
v1_series_acquisition 24 → 2.7 ms warm
Cold-pass numbers unchanged — the view runs the full pipeline on a
cache miss. Real-world OPDS traffic is dominated by warm hits.
Cross-user isolation verified manually: User A and User B (different
ACL) hit the same URL and receive different payloads (16 223 B vs
15 217 B); each user's warm response matches their own cold
response; `Vary: Accept, Cookie, Authorization, origin` confirmed
on every response.
Captured `stage2-before.json` + `stage2-after.json` alongside
`stage2.md` for the writeup.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* disable discord notification steps in gha
* OPDS views perf: Stage 3 — preview pipeline cache sharing + book queryset joins (#596)
Closes Tier 1 #4 (preview-pipeline re-runs) and the related #17 /
sub-plan 02 #3 (`select_related` shortfall on the OPDS book
queryset).
- `OPDS2FeedLinksView.get_book_qs` overrides the parent
`BrowserView.get_book_qs` to add `select_related("volume",
"language")`. The base method joins `series` only, with an
explicit comment that "OPDS doesn't need volume" — but
`Comic.get_title(volume=True)` reads `obj.volume.name` /
`obj.volume.number_to` per publication, and `_publication_metadata`
reads `obj.language.name`. Without these joins, every publication
iteration fired one lazy `Volume.objects.get` and one lazy
`Language.objects.get`. v1 already does the same join in
`v1/facets.py:64`; this brings v2 to parity.
- `_get_publications_preview_feed_view` shares `_admin_flags` and
`_cached_visible_library_pks` with the parent view. Both are
request-scoped (depend on user, not on params/kwargs), so it's
safe to skip the per-preview re-fetch. Cuts the visible-library
ACL lookup from 1-per-preview to 1-per-request.
`v2_start`: 53 → 29 cold queries (-24, ~45% reduction). Pre-fix
breakdown: 15 codex_volume (N+1) + 12 codex_library (4 per preview
× 3) dominated. Post-fix: 0 codex_volume + 3 codex_library.
Out-of-scope hotspots still visible in the trace (documented in
stage3.md): per-preview age-rating-with-metadata join (4 queries,
needs the bigger UNION-batch rewrite) and the 9 codex_comic filter
/ annotation queries (preserved as the legitimate per-preview
pipeline cost).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 4 — v1 acquisition M2M batching + progression PUT conflict detection (#597)
Closes Phase E from `tasks/opds-views-perf/99-summary.md`.
#7 — Per-page batching of v1 entry M2M fan-out
==============================================
Three new helpers in `codex/views/opds/metadata.py`:
- `get_credit_people_by_comic` — one query for all credits across
all comics on the page; partitions by comic_id in Python.
- `get_m2m_objects_by_comic` — UNION ALL across the 7
`OPDS_M2M_MODELS` tables, partitioned by `(comic_id, kind)`.
`OPDS1EntryData` gains `authors_by_pk` / `contributors_by_pk` /
`category_groups_by_pk` optional dicts. `OPDS1FeedView._get_entries_section`
populates them when `metadata=True` and `key=="books"`. Per-entry
properties read from the dicts when present, fall back to the legacy
single-comic helpers otherwise (so facet entries / single-comic feeds
still work).
Result: 9 queries per entry × N entries collapses to 3 queries per
page. On the harness's "All Batman" series with `?opdsMetadata=1`
(106 comics), `v1_acquisition_with_metadata` drops from 817 cold
queries / 1585 ms to **20 cold queries / 154 ms** (~40× / ~10×) —
verified in a controlled full-feed-state run.
#8 — Progression PUT atomic conditional UPDATE
==============================================
Two changes:
1. `OPDS2ProgressionSerializer.modified` flips from
`read_only=True` to `required=False`. Previously the field was
silently dropped from PUT validated_data, making the conflict
pre-check at `view.py:207-217` unreachable (zero
progression-related queries fired on PUT today).
2. `OPDS2ProgressionView.put` replaces the dead pre-check (which
was `_get_bookmark_query() + qs.first()` — never executed) with
a single atomic conditional UPDATE keyed on
`updated_at__lte=new_modified`. If the UPDATE matches a row,
write succeeds in one query. If 0 rows match AND a bookmark
exists, the DB has a fresher row → 409. If 0 rows match AND no
bookmark exists, fall through to the existing async
`update_bookmark` path (first-time write).
Behavior change: clients that previously sent stale `modified` and
got silent 200s now correctly receive 409 per the OPDS v2
progression spec. Functional verification:
- PUT no `modified` (no bookmark) → 200 (liberal accept, async create)
- PUT with stale `modified` → 409 (atomic conflict detection)
- PUT with fresh `modified` → 200 (atomic UPDATE)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* bump news for progression bugfix
* OPDS views perf: Stage 5 — Tier 3-4 cleanups (#598)
Closes Phase F from `tasks/opds-views-perf/99-summary.md`. Three
items land; two are intentionally skipped after audit.
#16 — `select_related("parent_folder")` on manifest queryset
`OPDS2ManifestView.get_object` adds the join; eliminates the
lazy `Folder.objects.get` per request when folder_view is on.
v2_manifest cold drops 23 → 22 queries.
#11 — Memoize filters JSON via `self.params["filters"]`
`_subtitle_filters` previously re-parsed `request.GET["filters"]`
inline (urllib.parse.unquote + json.loads). The same JSON is
already parsed by BrowserSettingsFilterInputSerializer; reading
from `self.params` skips the third parse. Sub-ms per request,
but on a hot client refresh cumulative.
#10 — Resolve `opds:bin:page` URL once for `_publication_reading_order`
Replace per-page `self.href()` (which fires `reverse()` each call)
with a single sentinel-page resolution + `str.format` substitution.
Saves N-1 `reverse()` calls per manifest hit. Invisible on the
harness's 1-page comic_pk=10785; visible on high-page-count PDFs.
#12 (audited won't-fix): the rescan is functionally necessary —
preview-group mtimes aren't covered by `_get_group_and_books`'s
mtime; removing the rescan would lose preview mtime tracking.
#18 (audited won't-fix): Stage 4 already provides the cleaner
SimpleNamespace pattern; legacy `_add_url_to_obj` fires only on
cold fallback paths where the cachalot reuse risk doesn't apply
(materialized list, not a queryset reused for caching).
After Stage 5 the OPDS perf project has reached the point where
remaining open items either need production telemetry (R3, #19),
medium-risk correctness verification (#9), or are negligible wins
versus framework cost (#14). Recommending pause until production
data points to a specific path.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Reader views perf plan: methodology + 3 sub-plans + ranked backlog (#599)
Audit of `codex/views/reader/` (~850 LOC across 6 source files)
mirroring the OPDS / browser-views perf plan structure. Identifies
15 ranked items + 5 research questions across three view families:
- Reader view chain (`c/<pk>` GET) — params/arcs/books/reader.py
- Reader settings (`c/settings`, `c/<pk>/settings`) — settings.py
- Reader page binary (`c/<pk>/<page>/page.jpg`) — page.py
Top three findings:
1. Comicbox archive open on every page request (sub-plan 03 #1) —
200-page comic = 200 archive opens per read-through. The biggest
single hotsp…
ajslater
added a commit
that referenced
this pull request
May 4, 2026
* update deps
* v1.10.1 fix opds v2 links
* fix news
* fix gha trigger gate logic
* fix gha yaml bug
* fix gha build and deploy from firing on develop
* fix admin update user view & serializer to allow user updates.
* fix gha yaml syntax error'
* fix runing build on develop branch
* fix gha branch gate logic again
* skip tests on deploy if identical files. change discord colors & messages. move ci contaiiner to compose
* fix polling all libraries when no ids submitted in task
* fix poll all library button
* fix opds v2 manifest series link
* force browser reset on start page of opds v2
* fix dev csp for vite hmr
* working server folder picker with enter activates menu and waits to render so the menu isn't lost in the center of the screen
* clean up server folder code to be smaller and remove cruft
* bump news
* attempt to make granian auto reload not hold on to file handles by adjusting template settings in debug mode
* Squashed commit of the following:
commit c76660006840abb36aa37d7355d5c7e242babebf
Merge: b2b5a011a be94a0ae0
Author: AJ Slater <aj@slater.net>
Date: Sun Apr 5 15:49:11 2026 -0700
Merge branch 'develop' into select-multiple
commit b2b5a011ab207a7307505b092877f789e1a66ab3
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:15:54 2026 -0700
fix card menu hover highlighting
commit 5a98fe23d89be5e11fa46ada927100ccd3e08cda
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:13:37 2026 -0700
new design for select many mode. no settings drawer involvement. reconfigure cards
commit 82c612e3f77eec90b02465174b49fc8552871686
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:50:11 2026 -0700
add github config to source include. remove dockerhub config
commit b02bd450f4598300d78433a2d749741872d14894
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:42:32 2026 -0700
update deps
commit d7bbc875e7936d808259c9ba726127fcc3af905e
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 23:56:12 2026 -0700
select many toolbar
commit 246d5ecda59aaabb83e41db3ad32ec552804b9ae
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 22:23:24 2026 -0700
select many feature first attempt
* bump news
* efficiency and bugfixes for gha yaml
* fix container name for gha
* fix test report steps running if tests don't run
* use eslint_d for frontend lint
* make sure to build the ci compose locally
* fix too exuberant volumes mounting for gha compose
* remove working dir entry i'm not sure it's good
* add debug line to gha workflow
* remove debug line
* fix test make order
* stop docker compose when done in gha
* i don't think i need DJGNO_SETTINGS_MODULE th ci thing
* fix down call to own bring down ci task
* fix docker compose down"
* fix test results upload
* fix test step name
* fix gha permissions
* bump news
* update deps
* browser set_params method saves last route and saves params to settings
* bump version to 1.10.4
* modify gha discord notification
* use happy-dom for frontend tests
* docker tag script better than the one in ci/
* update deps
* far saner param initialization for browsers and opds start pages
* update dpes & bump alpha version
* v1.10.5a0 (#551)
* bump news
* cirlcle ci no longer handles pre-release
* remove alpha scripts for circleci"
* fix lint-ci
* fix pre-relase gha
* fix default last route on start pages
* bump version
* fix default params
* alpha2
* regular version 1.10.5
* remove develop circleci builds
* adjust variable name
* bump version to 6 alpha 0
* fix default params for feed_views in opds 2
* debug logging for django crash
* v1.10.6a0 (#553)
* bump news
* ignore ruff qa for debug build
* ignore shellcheck"
* lint
* minor refactors of opds v2 feed
* regular v1.10.6 version
* fix docker-tag-latest cscript
* ignore gh token file
* try to log more request errors
* reconfigure logging to hopefully be more verbose about request errors in production
* update deps and bump to alpha version
* bump news (#555)
* v1.10.7
* bump news
* update devenv
* update devevn and deps
* fix pm script
* move django-check to test category
* workflow build frontend and collect static for prodcution build. fix test upload
* bump version and news 1.10.8
* fix dev-module script
* fix news
* explain news
* fix opds clear search setting
* fix clear search button
* use a registry cache instead of gha cache for the dist-builder
* gha use more env vars for image names. retain python dist for 2 days.
* new quick deploy gha script. update deps & devenv.
* silence watchfiles 5 second timeout debug message
* consolidate null values const
* make scope private
* update devenv
* update deps. migrate to unhead v3
* update deps
* fix creating reader global settings
* fix caching
* rename codex build-dist to codex-ci
* fix image name. make gha steps depend on each other more.
* fix gha syntax errors
* names for gha steps
* use ghcr.io for python-debian base
* update deps
* format dockerfile
* picopt treestamps
* fix custom covers not importing. v1.10.11
* fix custom covers count in admin view
* bump news for custom cover count fix
* update deps
* codex identification in server tag and opds generator tag
* update deps
* force no entries on opds start page
* common opds start page mixin. emtpy group objects on start page
* update deps. typechecking.
* api change q to search
* standardize search param as 'search' instead of 'q' or other variations
* remove errant icecream
* clear settings on backend
* Squashed commit of the following:
Fix clear settings null bug, add global settings clear button
- clearComicSettings was setting book.settings to null, causing
Object.entries() to throw TypeError downstream in getBookSettings
- Add null guard in getBookSettings as defense-in-depth
- Add null guard in isClearDisabled computed
- Add clearGlobalSettings action and clear button to Default Settings panel
- Compare against READER_DEFAULTS to determine if global clear is disabled
* simplify settings class hierarchy
* rename select-many store to browser-select-many
* switch to bun. updated devenv
* add a claude md
* use frozenattrdict to speed up configuration
* fix rename of browserSelectMany store
* auth token help
* fix sort-ignores to make deterministic across shells with different locales
* fix crash on settings not being raw
* another gaurd for getMetadta()
* remove keys from unhead meta headers
* fix unhead description for admin tabs"
* fix overzealous lazy importer
* fix lazyImportEnabled variable in metadata-activator
* fix metadata activator from bad cherry pick
* fix errant quote
* fix typechecking
* update devenv & deps
* bump news
* fix import bug linking folders
* fix possible batching crashes. adjust import variables for throughput
* batch comic updates
* move INTERNAL_IPS setting to general django area
* fix typechecking issue
* update deps
* bump version to v1.10.12
* fix redirect on OPDS alternate view with metadata
* minor change to browser empty page for better first time experience
* allow browsing to comics with any top group in opds
* bring project up to speed with bun and no package-lock.json
* fix browser paginator
* bump news for browser paginaor fix
* fix search combobox clearing
* uppercase book close button
* version v1.10.13. update deps
* fix pdfs not displaying
* fix csp for pdfs
* fix OPDS FK constraint failure when session row is missing (#607)
iOS Panels (and other Basic-Auth OPDS clients) intermittently hit
sqlite3.IntegrityError: FOREIGN KEY constraint failed when settings
or bookmarks were saved. Two interacting bugs caused it:
- Janitor cleanup_sessions used `if not session.get_decoded():` to
detect "corrupt" sessions. get_decoded() returns {} for both real
decode failures and legitimate anonymous sessions with no stored
data — exactly what Basic-Auth OPDS clients produce. The nightly
task was wiping valid session rows. Replaced with a direct
signing.loads() call so only genuine signature/decode failures are
flagged.
- _ensure_session_key returned the cookie's session_key without
verifying the row still exists. With cached_db the session loads
from cache without rechecking, so a stale cookie key would slip
through and cause an FK violation when used as SettingsBrowser /
SettingsReader.session_id. Now we verify existence and flush+save
to cycle the key when the row is gone.
Either fix alone closes the user-visible error; both together also
stop the underlying churn that created the bad state.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* extend session-key validation to bookmark + reader-settings paths (#608)
The same stale-session_key FK-violation pattern existed in two more
places that also write rows whose session FK can be stale:
- BookmarkAuthMixin.get_bookmark_auth_filter — feeds session_id into
Bookmark.objects.bulk_create / bulk_update.
- ReaderSettingsBaseView._get_bookmark_auth_filter — feeds session_id
into SettingsReader.objects.create.
Both used the old `if not session.session_key: save()` pattern that
trusts the cookie. Hoist the validated _ensure_session_key helper
from SettingsBaseView up to AuthMixin so every auth-aware view shares
one implementation, and switch both call sites to it. BookmarkAuthMixin
now extends AuthMixin to inherit the helper. BookmarkFilterMixin is
unchanged — it's read-only (filter Q only) and a missing session
correctly returns no rows.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* format
* bump news
* Stats: fix user_registered_count / auth_group_count always-zero bug (#611)
The /admin/stats endpoint's user_registered_count and
auth_group_count fields have been silently returning 0 since at
least Sep 2024. The Stats tab in the admin UI shows 0 registered
users even on installs with multiple accounts.
Root cause: _add_config tried to rename the per-model count keys
produced by _get_model_counts:
config["user_registered_count"] = config.pop("users_count", 0)
config["auth_group_count"] = config.pop("groups_count", 0)
But _get_model_counts builds keys via
``snakecase(model.__name__) + "_count"``. For Django's
``django.contrib.auth.models.User`` / ``Group`` that produces
``user_count`` and ``group_count`` (singular). The pop()s with the
plural names never matched, so the default ``0`` won every time —
and the actual ``user_count`` / ``group_count`` keys were left
orphaned in the dict, then dropped by the StatsConfigSerializer
which only declares ``user_registered_count`` /
``auth_group_count``.
Fix: pop the right source keys.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* defaults for dockerfile ARGs
* format'
* Frontend correctness: 10 bug fixes from sub-plan 01 (#647)
* Reader page: fix load-progress spinner that never appeared
Two stacked bugs in the same setTimeout:
1. Non-arrow callback lost ``this``. The function ran with the
timer's context, not the component's, so ``this.loaded`` and
the write below were both no-ops.
2. The write targeted ``this.loading``, which has never been a
data field on this component. The template binds the spinner
to ``showProgress`` (line 15: ``v-if="showProgress && !loaded"``).
So even if the arrow had been there from the start, the spinner
still wouldn't have rendered — both bugs had to land at once.
Net: ``LoadingPage`` has been dead code for slow image loads.
Switch to an arrow function and write ``showProgress`` instead
of ``loading``. Stash the timer ID so ``beforeUnmount`` can
clear it; a fast page swap mid-delay would otherwise fire the
write on a torn-down component.
Implements B1 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: fix arc-mtime fallback that itself 500'd
``loadMtimes`` builds an arcs list of ``{ group, pks }`` from
``this.arcs``; if the dict is empty the function previously
fell back to ``arcs.push({ r: "0" })``. The comment noted that
"No arcs is a 500 from the mtime api" — the fallback was added
to dodge that 500 — but the wrong-shape fallback also produced
a 500 because the API expects ``group``/``pks`` keys, not ``r``.
Use the canonical shape so the fallback actually works.
Implements B2 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* iOS PWA download: pass the object URL to revokeObjectURL, not the Blob
``URL.createObjectURL(blob)`` returns a ``blob:...`` URL string;
``URL.revokeObjectURL`` must receive that same string to free the
mapping. The previous code passed ``response.data`` (the Blob
itself), which silently no-op'd and leaked one object URL per
download.
On iOS PWAs this matters more than elsewhere because the leak
accumulates across the user's session and can't be reclaimed
short of reloading the app.
Implements B6 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser API: stop mutating caller's settings in getGroupDownloadURL
``getGroupDownloadURL`` did ``delete settings.show`` on the
caller's object before building the URL. Side-effect: any caller
that re-used the settings dict after the download-URL build saw
its ``show`` key silently vanish. This was probably fine when
the function was first written but it's a footgun now that
settings flow through a Pinia store.
Destructure-and-spread to drop ``show`` without touching the
input.
Implements B8 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Auth store: make logout awaitable + clear state unconditionally
Two changes to the logout action:
1. ``async`` so callers can ``await``. The current call site
(``auth-menu.vue``) fires and forgets, but a future UX pass
that wants to disable the button while logout is in flight
needs the promise.
2. Clear ``this.user`` in ``finally`` rather than only on
success. The user clicked "log out" — UI should reflect the
logged-out state immediately, regardless of whether the
server-side logout endpoint succeeded. Server-side cookies
that survive the network failure will get cleaned up by the
next 401.
Implements B7 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser filter menu: stop double-rendering each filter row
``<v-list>`` was passed both ``:items="vuetifyItems"`` AND a
default-slot ``v-for`` over the same list. Vuetify renders the
items prop into ``v-list-item`` children directly, so every row
was being built twice — once by the prop, once by the manual
``v-for``. Visible to users on filter menus with large choice
lists (genres, characters, etc.); each row appeared duplicated
and the DOM cost doubled.
Drop the prop. Keep the ``v-for`` because it carries the custom
``#append`` slot for ``metronName`` rendering. ``:model-value`` /
``@update:selected`` still drive selection state via each list-
item's ``:value`` prop.
Implements B10 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Admin job-tab: remove API-fetching click handler from expanded panel
The expanded status panel had ``@click="loadAllStatuses"`` on its
container div. Any click inside the panel — including clicks on
child elements that bubbled — refetched the entire status map.
Probably copy-pasted as a "refresh on click" gesture, but it
fired far too often: a user inspecting a long status list would
trigger N API calls just from glancing around.
The status data is pushed through the websocket already
(socket.js fans librarian notifications into the admin store),
so the panel is up-to-date without a manual refresh.
Implements B11 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: handle bookmark-write errors instead of silently rejecting
``setRoutesAndBookmarkPage`` awaited ``_setBookmarkPage`` but
didn't catch its errors. On a network blip the promise rejected,
the bookmark didn't persist, and the failure became an unhandled
rejection in the browser console — not visible to the user, not
retried, just lost.
Wrap in try/catch. The local page state stays where it is (the
user is reading forward; the bookmark catches up on the next
write), but the failure is logged so debugging surfaces. A
proper user-visible toast + retry path is broader UX work
tracked in the plan.
Implements B3 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: clear progress timer on unmount
``updateProgress`` chains itself via ``setTimeout`` until the
metadata loads or progress reaches 100. The timer ID was never
stashed, so closing the dialog mid-animation left the chain
running — each tick fired on a torn-down component, writing
``this.progress`` and re-scheduling against now-null refs.
Stash the timer ID and clear it in ``beforeUnmount`` so the
chain stops cleanly when the dialog goes away.
Implements B12 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader pager: key dynamic component on identity to force remount
``<component :is="...">`` without ``:key`` lets Vue reuse the
existing instance across an ``is`` change when the components
share enough surface (props, name). For the reader's
vertical/horizontal pager swap that's wrong: scroll listeners
attached by the previous mode persist, the new mode's
``mounted`` runs against stale internal state, and any
abort/teardown logic in ``beforeUnmount`` never fires.
Add ``:key="component.name"`` so the swap is a true unmount +
remount — old listeners go away, new mode starts clean.
Implements B13 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: lint cleanup for the B12 fixup
Vue option-order rule: ``beforeUnmount`` belongs above
``methods``. Block-comment style required for the multi-line
explanation in ``updateProgress``. Both surfaced when running
eslint on the prior commit; pure cleanup, no behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* news for cherry pick
* OPDS auth: send WWW-Authenticate + opds-authentication content-type on 401 (#652)
Panels and other strict OPDS clients require the WWW-Authenticate header
(per RFC 7235) to trigger the auth prompt, and the spec calls for
application/opds-authentication+json on the auth document. The exception
handler was already converting 403 to 401 with the auth doc body, but
was bypassing DRF's natural 401 response and dropping both the header
and the proper content type, which Panels read as a forbidden state.
Also fix a stray `from re import DEBUG` in the auth view that always
forced the absolute-URL path.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix typing inheritence with drf perms
* bump version to v1.10.14
* update picopt treestamps
* fix typing import error
* fix vulture ignorelist
* readd the mistakenly removed vulture_ignorelist
* v1.10.15 fix show order bug. update deps
* fix nightly job manual expansion
* fix news version number
* V1.11 performance (#714)
* title for age rating tagged
* rename unrestricted to adult for clarity
* remove circleci code
* update to new devenv scripts in frontend
* fix sort-ignores to make deterministic across shells with different locales
* batch comic updates
* fix possible batching crashes. adjust import variables for throughput
* bump news for v1.11.0
* Add features to readme
* add saved view feature
* Add browser-views perf measurement harness (Stage 0) (#574)
Stand up the measurement infrastructure that gates the rest of the
browser-views performance work:
- Add django-silk 5.5 to the dev group; install + route it to a
separate ``silky`` SQLite DB via a new ``SilkRouter`` so profiler
traces never touch the live DB.
- Wire SilkyMiddleware below ServeStaticMiddleware so it only wraps
the API stack, not static-file responses.
- Expose /silk/ under the existing DEBUG URL block.
- Add ``tests/perf/run_baseline.py``: hits the live dev DB via
``django.test.Client``, runs three cold/warm flow pairs (root
browse, filtered search, series metadata), and writes a JSON
baseline artifact. Cachalot + page-cache are invalidated before
each cold pass so the numbers reflect actual DB work.
- Add ``make perf-baseline`` target.
- Commit the per-view analysis docs and initial baseline capture
under ``tasks/browser-views-perf/`` for reference during the
follow-on cleanup stages.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 1 - startup + annotation + filter micros (#575)
Bundle of small, surgical wins observed on the slimlib baseline:
- Cache `libraries_exist` with a 60s TTL in Django's default cache;
invalidate on Library save/delete via lazy-imported signal handlers.
Drops a redundant `Library.objects.filter(...).exists()` from every
browser response.
- Short-circuit the four `_save_browser_*` code paths in
`codex/views/settings.py` when nothing actually changed. Avoids
gratuitous `.save()` calls that flush cachalot on no-op PATCH writes.
- Dedupe the `add_group_by(qs)` call in
`codex/views/browser/browser.py`: group once in
`_get_common_queryset` (no-op for Comic), drop the second call in
`_get_group_queryset`.
- Memoize `get_max_bookmark_updated_at_aggregate` per
`(model, agg_func, default)` on the view instance — the three
callers (group_mtime, order, bookmark) now share one Aggregate.
- Move the `bmua_is_max` flag off the per-row `Value` annotation and
read it from `self.context["view"]` in the browser serializer.
- `@lru_cache(maxsize=256)` on `_preparse_search_query`: extract to a
pure module-level helper keyed on `(text, path_allowed)`; returns
frozen tokens.
- `@lru_cache(maxsize=512)` on `get_field_query`: cache the Lark-parsed
Q tree, `copy.deepcopy` on return because `_hoist_filters` mutates
`child.negated` downstream.
- Stash the `BaseDatabaseOperations(None)` singleton as `_DB_OPS` in
`search/field/expression.py`; `prep_for_like_query` doesn't use the
connection.
- Pre-filter the field loop in `filters/field.py` to keys actually set
in the request — saves ~20 no-op calls per browser.
- Delete `search/field/optimize.py` (`like_qs_to_regex_q` and friends
were already unused — grep confirms only self-references).
Slimlib cold baseline (3 flows, stage1.json vs Stage 0 baseline.json):
- flow_a_root_browse: 21→18 SQL, 189.6→182.3 ms
- flow_b_filtered_search: 21→17 SQL, 187.4→178.0 ms
- flow_c_series_metadata: 34→31 SQL, 251.1→226.9 ms
Warm paths unchanged (0 SQL, ~2 ms). Full tests + ruff pass.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix type warnings
* fix lint error
* format
* Run silk migrations on silky DB at startup (#576)
Stage 0 wired a second SQLite DB (silky) for django-silk profiling traces
and a DATABASE_ROUTERS entry that routes silk app models there. But
ensure_db_schema only invoked `call_command("migrate")` without a database
arg, which only migrates the default DB. The router then blocks silk
migrations from running on default — so the silky DB was never populated,
and the first request through SilkyMiddleware failed with
"no such table: silk_request".
Mirror the pattern from tests/perf/run_baseline.py: after the default
migrate, also run `migrate silk --database=silky` when the silky DB is
configured. Guarded on `"silky" in connections.databases` so production
(DEBUG=False, no silky DB) is a no-op.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 2 — triple COUNT + page mtime cache (#577)
## PR 2a — Eliminate triple COUNT on the paginate path
Each browse request ran three COUNT queries per section (groups & books):
1. The outer grouped COUNT in `_get_common_queryset`.
2. Paginator's internal COUNT triggered by `paginator.page()`.
3. An explicit `.count()` on the paginated slice.
The outer COUNT is needed (sizing the paginator). The other two are
redundant — the page row count is bounded by `per_page` and derivable
from `end_index - start_index + 1`.
- Shadow `Paginator.count` (a `@cached_property`) with the pre-computed
total to skip Paginator's internal COUNT.
- Derive page row count arithmetically from `Page.start_index()` /
`end_index()`.
- Pass `book_count` through `paginate()` alongside `group_count`; drop
`book_qs.count()` on the opds2 path.
- `_paginate_section` returns `(qs, count)` directly.
Short-circuits on `total_count == 0` (avoids Paginator instantiation on
empty sections) and preserves the EmptyPage warning branch.
## PR 2b — Short-TTL page mtime cache
`BrowserView._get_page_mtime()` calls `get_group_mtime(page_mtime=True)`
on every browse request. The query is a filtered Max aggregate that
cachalot caches — but any write to Comic / Bookmark invalidates it, so
bookmark-heavy usage forces recomputation. Cold-path silk traces show
this aggregate at ~26ms on flow_a — the second-slowest query in the
request.
Add a 5s TTL cache layer gated on page_mtime=True. Key includes user,
model, group, pks, page, and a hash of filter-affecting params
(filters, search, q, order_by, order_reverse). The polling MtimeView
path (no page_mtime) is unaffected, so frontend change-detection stays
live.
## Measurements
tests/perf/run_baseline.py on the slimlib DB. Cold = full cache
invalidation; warm = cachalot populated.
| flow | stage1 cold | stage2 cold |
|-------------------------|------------------------|------------------------|
| flow_a root browse | 18 queries / 182.3 ms | 16 queries / 135.6 ms |
| flow_b filtered search | 17 queries / 178.0 ms | 15 queries / 130.3 ms |
| flow_c series metadata | 31 queries / 226.9 ms | 31 queries / 229.9 ms |
flow_a / flow_b: -2 queries, ~26% cold wall-time reduction. flow_c
unaffected (metadata doesn't traverse paginate). PR 2b's benefit is a
dogpile guard after cachalot invalidation — doesn't show in the harness
(cold = both caches empty; warm = cachalot wins first).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix browser paginator
* typecheck browser paginate
* Browser views perf: Stage 3 — cover fan-out collapse (#578)
* Stage 3 — cover fan-out: per-pk endpoint + pre-resolved cover_pk
Before stage 3, every cover card triggered a full BrowserAnnotateOrderView
pipeline to pick the representative comic pk. A default page does ~100 of
these in parallel. This PR collapses that into one correlated Subquery on
the browse response plus a thin per-pk cover endpoint.
Components
- Option A: annotate_order_aggregates(for_cover=True) drops the JsonGroupArray
+ page_count aggregates from the cover path — unused for picking a pk.
- Option B.1: BrowserView group cards get cover_pk / cover_custom_pk via
correlated Subquery that replicates CoverView.get_group_filter exactly
(direct fk match when dynamic_covers/Volume/Folder; sort_name fuzzy match
otherwise, correlated on _GROUP_BY columns so the same comic set as `ids`
is picked — no JSON parsing, no peer aggregate).
- New endpoints /api/v3/c/<pk>/cover.webp and /api/v3/cc/<pk>/cover.webp
serve already-resolved pks with a cheap single-row ACL probe and the
existing CoverPathMixin / CoverCreateThread pipeline.
- Frontend getCoverSrc prefers the new per-pk URL when cover_pk /
cover_custom_pk is on the card; falls back to the old group+pks URL
otherwise, so OPDS and search-active browses keep working.
- FTS skip: cover_pk annotation is skipped when params["search"] is set.
MATCH inside a correlated subquery re-scans the FTS5 index per outer
row (~900ms on a 100-group page). The old URL path still applies the
search filter per cover — same behavior, parallelized over HTTP.
Perf (slimlib dev DB, Flow D = browse + every card's cover):
Flow Before cold After cold Delta
A root browse 156.9ms / 15 q 180.5ms / 15 q +24ms / +0 q
B search 148.3ms / 16 q 132.4ms / 16 q -16ms / +0 q
C metadata 161.9ms / 31 q 165.0ms / 31 q +3ms / +0 q
D browse+covers 2311ms / 1216q 1161ms / 815 q -50% / -33%
Flow A takes a mild regression for the correlated cover subquery, but Flow
D — the realistic user wall-clock — drops by half and 400 SQL queries.
Subsequent cover fetches drop from ~90ms each (full pipeline) to ~5ms
(disk read + 1 ACL probe).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Stage 3 follow-ups: custom_cover URL, FTS pre-materialization, OPDS thin covers
Address review feedback on the cover fan-out collapse:
- Rename /api/v3/cc/<pk>/cover.webp → /api/v3/custom_cover/<pk>/cover.webp.
Descriptive over terse; matches the view name.
- FTS pre-materialization replaces the FTS-skip fallback. A correlated
MATCH re-scans FTS5 per outer row (~900ms on a 100-group page); the
old path worked around this by skipping cover_pk annotation on search
and sending every cover through the legacy pipeline once. We now
pre-select the FTS match set as a non-correlated sub-SELECT in the
outer cover subquery — SQLite materializes it once and each correlated
cover row lookup becomes an indexed pk filter. Cover_pk is annotated
on search responses, the thin endpoint handles each cover.
- OPDS now emits thin-endpoint cover URLs (v1 and v2):
- New OPDSComicCoverByPkView / OPDSCustomCoverByPkView wrap the browser
thin views with OPDSAuthMixin's Basic Auth.
- New opds:bin:cover_by_pk and opds:bin:custom_cover_by_pk URL names.
- v1 _cover_link picks: cover_custom_pk → custom thin; group=='c' → own
pk thin; cover_pk → thin; else legacy group+pks fallback.
- v2 _thumb always uses the thin endpoint (publications are Comic rows
so obj.pk IS the cover pk).
- OPDS inherits annotate_cover() via BrowserView._get_group_and_books,
so group rows already carry cover_pk / cover_custom_pk.
- tests/perf/run_baseline.py gains a flow_e (search + covers) to measure
the search path end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix search combobox clearing
* format
* Stage 3 follow-ups: folder covers, endpoint rename, offline cover pipeline (#579)
- Fix folder cover_pk picking comics not actually in a folder's subtree
by using the recursive ``folders`` M2M (same relation the browse filter
uses) for the per-card cover subquery instead of the direct
``parent_folder`` FK.
- Delete the legacy thick cover endpoint. Rename cover_by_pk.py to
cover.py and drop the "_by_pk" suffix from view class names
(CoverView, CustomCoverView, OPDSCoverView, OPDSCustomCoverView).
Update URL configs, OPDS wrappers, and the frontend client.
- Move all cover generation off the HTTP path. Add CoverCreateTask and
enqueue it from the importer after bulk_create so new comics get
thumbnails pre-warmed offline. When a cached thumb is missing the
per-pk endpoint enqueues the task and responds 202 Accepted with
Retry-After plus the missing-cover placeholder instead of synthesizing
the WebP inline under a worker thread.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update devenv sort ignores
* Force cover refresh after 202 so the placeholder doesn't stick (#580)
Browsers don't honor Retry-After on <img> elements and happily cache the
placeholder served with the 202 response, so even after the cover thread
finishes writing the real thumb the img src keeps rendering the stale
placeholder bytes.
- Backend sends Cache-Control: no-store on the 202 placeholder so the
response isn't cached at that URL.
- BookCover now probes the cover URL with fetch() on mount and, if it
sees 202, waits Retry-After and bumps a reactive `retry` counter that
becomes a cache-busting query param on coverSrc. v-img re-fetches with
the new URL and gets the real cover once the cover thread is done.
Retries are capped at 5 and aborted on unmount via AbortController.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Harden cover endpoint against concurrent read/write races (#581)
Under a full cold-cache page load, 32 HTTP workers hitting the cover
endpoint at the same time as the cover thread was writing thumbs
returned 500s for a handful of covers. The endpoint used a three-step
exists()/stat()/read_bytes() sequence that could race against an
in-flight write, and save_cover_to_cache wrote directly to the target
path so readers could observe a truncated file mid-write.
- save_cover_to_cache now stages to a ``{name}.{pid}.tmp`` sibling and
renames with Path.replace, so a read either sees the pre-existing
state or the fully written file — never a partial one. Stale temps
are unlinked on failure.
- _get_cover_response collapses to a single read_bytes() that catches
FileNotFoundError and logs OSError, then falls through to the 202
path. No race window between stat and read.
- LIBRARIAN_QUEUE.put and the outer get() methods are wrapped in
try/except that log and return the placeholder, so a surprise
exception is never user-visible as a 500 and always leaves a log line.
- cleanup_orphan_covers runs a dedicated ``*.tmp`` sweep over both cover
roots before the orphan scan, so stale temps from crashed writers
don't accumulate.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix custom cover generation
* relegate watch for changes restart to a en env variable settings
* await and finish watch task
* update devenv
* Fix metadata cover fetching pk=0 after Stage 3 cover fan-out (#583)
Stage 3's follow-up commit (2ff08524) dropped the legacy `group+pks`
fallback from the frontend `getCoverSrc()` helper, leaving it to only
build per-pk URLs from `coverPk` / `coverCustomPk`. The browser card
pipeline annotates those fields via `BrowserAnnotateCoverView`, but the
metadata endpoint was never wired into that annotation, and
`metadata-cover.vue` never passed the cover pks through to `BookCover`.
As a result every metadata dialog tried to load
`/api/v3/c/0/cover.webp` — pk 0 — and fell back to the missing-cover
placeholder.
Wire the annotation + serializer + component end-to-end:
- `MetadataAnnotateView` inherits from `BrowserAnnotateCoverView` so
`annotate_cover()` is available on the metadata queryset.
- `MetadataView.get_object()` calls `annotate_cover(qs)` after
`annotate_card_aggregates(qs)`, mirroring `browser.py`'s ordering.
- `MetadataSerializer` exposes `cover_pk` and `cover_custom_pk` as
`SerializerMethodField`s with the same fallback-to-`obj.pk` semantics
that `BrowserCardSerializer` uses — so Comic metadata (`group=c`)
works without annotation, falling back to its own pk.
- `metadata-cover.vue` forwards `md.coverPk` / `md.coverCustomPk` to
`BookCover`.
Verified on /api/v3/{p,i,s,v,c,f}/*/metadata: `coverPk` now resolves
to a real comic pk on every group, and the comic path falls back to
its own pk. No SQL query delta — `annotate_cover` is a correlated
Subquery that inlines into the outer SELECT.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hints (#582)
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hydration hints
Fold the three `_intersection_annotate()` passes (value fields, conflict
shadow fields, related-count fields) into a single batched call that
collapses N distinct-count probes into one `aggregate()` and one
`values_list()`, with unique synthetic keys so annotation groups don't
collide.
Extend `FK_QUERY_OPTIMIZERS` so intersecting FK hydration
(`AgeRating`, `Character`, `Team`) carries `select_related` + `.only()`
hints for the nested fields the metadata serializer actually reads
(`AgeRating.metron`, `Character.identifier.url`). Without this, each
nested access fired a follow-up query per instance.
For the M2M intersection path, short-circuit empty-intersection fields
with `Model.objects.none()` so we skip the optimizer setup (and, for
the Comic self-reference path, pointless prefetch dispatches on already
empty results).
Add `flow_c2_comic_metadata` to the perf baseline harness so the
comic-detail metadata path (the FK + M2M heavy flow) is tracked
alongside the series flow.
Measured: flow_c_series_metadata cold SQL drops from 31 → 28 (-3).
flow_c2_comic_metadata baseline captured at 47 queries for future
stages. Other flows unchanged within noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* WATCH for changes variable depends on debug
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Fix folder + FTS crash and preserve rank-ordered cover selection (#584)
* Fix folder browse + FTS crash on search_score ordering
Browsing folders (or any group) with an FTS search and
``orderBy=search_score`` blew up with::
sqlite3.OperationalError: no such column: codex_comicfts.rank
Root cause: Stage 3's cover fan-out collapse builds a correlated
Comic subquery per group card to pick ``cover_pk``. The follow-up
(#579) replaced the correlated ``comicfts__match`` filter with a
non-correlated ``pk__in fts_sq`` pre-materialization — so the Comic
subquery no longer joins ``codex_comicfts``. But
``annotate_order_aggregates`` still annotated
``search_score=ComicFTSRank()`` inside the subquery, and
``add_order_by`` issued ``ORDER BY "codex_comicfts"."rank" * -1``,
which SQLite can't resolve from the subquery's FROM list.
- ``_annotate_search_scores`` now takes ``for_cover`` and skips the
annotation when set. ``annotate_order_aggregates`` threads the flag
through (matching the existing ``for_cover`` pipeline-trims).
- ``_cover_comic_subquery`` passes ``order_key="sort_name"`` to
``add_order_by`` whenever the user's order is ``search_score`` — the
cover's tie-break isn't user-visible, and ``sort_name`` is a real
indexed column on Comic.
Verified: /api/v3/{r,f,a,p}/0/1?q=iron+man&orderBy=search_score all
return 200 with a real ``coverPk`` (not 0). Lint + 20-test pytest
suite pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Preserve FTS rank ordering in cover subquery
The previous fix short-circuited to ``sort_name`` when ``order_by`` was
``search_score`` so a group card's cover could be picked without
referencing ``codex_comicfts.rank``. That fixed the crash but meant the
cover chosen for each card was the alphabetically-first matching comic,
not the top-ranked FTS match — a UX regression on searched pages.
Restore rank-ordered cover selection by:
* Applying ``fts_q`` (``comicfts__match=...``) directly in the cover
subquery so ``codex_comicfts`` is joined and ``rank`` is populated,
while keeping the ``pk__in`` pre-materialization for cheap filtering.
* Teaching ``ComicFTSRank`` to resolve the query-local alias for
``codex_comicfts`` at compile time. Its literal template worked for
the top-level browse query but emitted an unresolvable column ref in
nested subqueries where Django aliases the join as ``V4``/``U1``.
* Skipping ``.group_by("id")`` in the cover-subquery search_score
annotation. The custom force-group-by compiler emits a literal
``"codex_comic"."id"`` that breaks under nested aliasing, and the
cover subquery is already ``.distinct() … LIMIT 1`` so dedup is
redundant there.
Verified against the dev DB: for each publisher card on a ``batman``
search, the annotated ``coverPk`` now matches the top-ranked comic
returned by a direct ``ComicFTSRank`` ordering within that publisher.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update for claude rules
* Browser views perf: Stage 5a — server-side cache_page on cover endpoints (#585)
Wrap /api/v3/c/<pk>/cover.webp, /api/v3/custom_cover/<pk>/cover.webp,
and their OPDS counterparts in cache_page(COVER_MAX_AGE). Compose with
cache_control + vary_on_cookie (API) or vary_on_headers("Cookie",
"Authorization") (OPDS, which accepts Basic + Bearer + Session auth)
so the Vary header is set before cache_page stores the response — the
cache key is keyed per auth identity, no cross-user leakage.
Also raise Django FileBasedCache MAX_ENTRIES from the default 300 to
10000. Cachalot query results + cache_page entries (browser + cover)
exceed 300 during a single browse-with-covers pageload, triggering the
2/3 random cull that silently evicts just-populated cover entries
before the next request can read them. Without this, Flow D warm only
dropped to 743 (~50% cover-cache hit rate); with it, Flow D warm drops
to 0.
Perf impact (stage5a-after.json vs. stage4-after.json):
Flow D — browse + 100 covers warm 802 → 0 queries
Flow E — search + 46 covers warm 368 → 232 queries
Flow E's residual 232 queries are 29 covers that return 202 Accepted
(cover not yet generated; response has Cache-Control: no-store, which
cache_page correctly skips). The 17 covers that returned 200 were 0
queries each. In production, 202s resolve within seconds as the cover
thread catches up, so steady-state warm on Flow E is also 0.
Cold query counts and Flow A/B/C are unchanged.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Raise RLIMIT_NOFILE on startup to avoid macOS 256 FD cap (#586)
Cold browser sessions loading a full 100-card grid intermittently crash
the dev server with `OSError: [Errno 24] Too many open files`. Diagnosis:
- macOS ships a 256 soft cap for RLIMIT_NOFILE (`ulimit -Sn 256`).
- Each Django request thread keeps a sticky SQLite connection
(`CONN_MAX_AGE=600`).
- SQLite WAL mode opens 3 FDs per connection (main + `-wal` + `-shm`).
- The thin cover endpoint dispatches to a `sync_to_async` threadpool, so
a burst of 100 cold cover requests easily spawns ~100 threads → ~300
SQLite FDs, blowing past the 256 cap before page reads even open.
Bumping the soft limit toward the hard cap (or 8192, whichever is lower)
at process start is non-invasive and matches what production deployments
typically achieve via shell `ulimit`. No-op on Linux (already high) and
on platforms without the `resource` module.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps, format json
* Browser views perf: Stage 5b — annotation gating + m2m-aware distinct (#587)
Three surgical correctness/perf changes that surface the right SQL for the
right target. The Stage 5a work (cover endpoint cache_page) collapsed Flow D
warm to 0; 5b cleans up cold-path waste the harness flows don't directly
exercise.
5.2 — Skip ``updated_ats`` JsonGroupArray outside browser/metadata
``obj.updated_ats`` is consumed only in
``BrowserAggregateSerializerMixin.get_mtime`` (browser + metadata
serializers). OPDS computes its own mtime from the bookmark aggregate;
cover/download paths never read it. Skip the DISTINCT scalar aggregate
for those targets.
5.3 — m2m-aware .distinct() in ``BrowserFilterView.get_filtered_queryset``
New ``comic_filter_uses_m2m`` cached property. Comic queries skip
``.distinct()`` unless a real m2m or m2m-through join is present
(story_arc browse, folder browse on cover/choices/bookmark/download
TARGETs, or any m2m field filter). Non-Comic queries still always
``.distinct()`` because the ACL alone traverses ``comic__``
(one-to-many).
5.5 — Tie ``search_score`` ``group_by("id")`` to the same flag
``annotate/order.py:_annotate_search_scores`` only emits the GROUP BY
when fan-out actually exists. Cover path stays gated by ``for_cover``.
5.4 — Skipped intentionally
``codex/views/auth.py`` already caches the three scalar inputs to
``get_acl_filter`` per request. The remaining cost is composing two Q
objects from those scalars — microseconds, not queries. Wrapping a
cached_property keyed on ``(model, user_id)`` would be cosmetic.
20 standalone cases of ``comic_filter_uses_m2m`` pass (every default-target
group, every m2m-folder TARGET, every BROWSER_FILTER_KEY by category, mixed
filters, ``pks=(0,)`` early-out).
stage5b-after.json: existing perf flows preserved (query counts identical;
wall times within run-to-run noise). The wins are on cold paths the harness
doesn't exercise — browsing a Series's comics, default-TARGET folder browse,
OPDS feeds — which a follow-up should add.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5c — batched choices_available (#588)
Replaces the per-field probe loop in BrowserChoicesAvailableView with a
single batched EXISTS annotate. FK fields keep "any non-null exists"
semantics; m2m fields decompose into (has_rel, has_null) booleans plus a
lazy distinct-count probe for the rare has_rel ∧ ¬has_null corner.
The natural EXISTS(SELECT DISTINCT rel ... LIMIT 1 OFFSET 1) form is
broken on SQLite — EXISTS short-circuits on the first row from the
underlying join, before DISTINCT collapses or OFFSET skips — so the m2m
path uses two cheap booleans + a Python-side cap-at-2 distinct probe.
Perf: flow_f_choices_available cold drops 34 → 11 queries (−68%) and
121 → 53 ms (−56%) on the dev DB. Other flows are noise-level
unchanged. tests/perf/run_baseline.py picks up three new flows
(choices_available, m2m field, FK field) so the changed code path is
visible in the artifact.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* run frontend with bun
* fix syntax error in filter sub menu
* Browser views perf: Stage 5d — has_metadata boolean cast + series-browse perf flow (#589)
Cleanup bundle (5.7) shrinks against post-5c code:
- #26: switch ``has_metadata`` annotation from ``F("metadata_mtime")`` to
``ExpressionWrapper(Q(metadata_mtime__isnull=False), BooleanField())`` in
both ``codex/views/browser/annotate/card.py`` and
``codex/views/reader/books.py``. Matches the consumer serializer's
``BooleanField`` and trims the SELECT projection to one byte per row.
- Harness: add ``flow_a2_series_browse`` so the harness covers the
Comic-queryset / no-m2m-filter path that Stage 5b's distinct + group_by
skip wins on. Headline measurement: 6 cold / 4 warm queries, ~13 ms cold.
Items #18 (already absorbed in 5c), #25 (parent-aware reduction already in
place; trimming further would change ORDER BY semantics), #30 (already
coalesced inside ``BookmarkUpdateMixin.update_bookmarks``), and #31
(``zipstream-ng`` already streams via ``FileResponse``) were re-investigated
and skipped with reasons documented in §10 of ``05-replan.md``.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff (#590)
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: 99-summary status column + OPDS perf plan (#591)
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: backlog status column on 99-summary
Stage 5 final exit criterion. Adds a Status column to every tier table in
tasks/browser-views-perf/99-summary.md so the backlog reads as a closed
ledger rather than an open plan.
Markers:
- Tier 1 (4): all four landed (Stage 2, 3, 4, 5a)
- Tier 2 (6): all six landed (Stage 1 / 5b)
- Tier 3 (7): four landed (Stage 2, 5b, 5c), one skipped (#16
subsumed by GroupACLMixin per-request scalar caches), three open
(#11, #12 absorbed by Stage 4 hints; #15 not on a hot Flow A-H path)
- Tier 4 (14): seven landed (Stage 1 / 5c / 5d), four re-investigated
in Stage 5d and confirmed already done (#25, #27, #30, #31), three
open (#22, #24, #29)
- Tier 5 (3): R1 ✅ Stage 5.9 (FTS demote regression test), R2 ❌
skipped (current code is correct), R3 ⏭️ deferred via the
stage5e-handoff-serializer-audit.md brief
Pure documentation update — no code or test changes. Closes the last
unchecked exit criterion on the Stage 5 plan.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: initial planning pass
Mirrors the structure of tasks/browser-views-perf/ for the OPDS surface.
No code changes — produces a ranked backlog so the same per-stage rigor
can be applied to OPDS once browser-views perf wraps.
Files:
- 00-meta-plan.md methodology, scope, sub-plan layout, exit criteria
- 01-routes-and-cache.md OPDS_TIMEOUT=0 disables cache_page on every feed
- 02-feed-pipeline.md v1 + v2 main feeds, BrowserView reuse, preview reruns
- 03-entry-serialization-v1.md v1 entries, lazy_metadata Comicbox open, M2M fan-out
- 04-publications-v2.md is_allowed static-method bypass of admin_flags cache
- 05-manifest.md credit fan-out 11→1, story_arcs N+1, subjects 7→1
- 06-progression-binary-aux.md PUT conflict pre-check + dead expr; binary inheritance
- 99-summary.md 5-tier ranked backlog, phasing A–F, cross-cutting guidance
Top findings (in landing order):
1. OPDS_TIMEOUT = 0 in codex/urls/const.py — every feed wraps
cache_page(OPDS_TIMEOUT) and gets nothing. Single-line config flip
is the highest-leverage win and gates Phase A.
2. Manifest credit fan-out (v2/manifest.py:194-199) — 11 separate
Credit.objects.filter queries because _MD_CREDIT_MAP is iterated.
Collapsible to one query + Python partition.
3. is_allowed static method (v2/feed/publications.py:35-56) bypasses
the request-cached admin_flags MappingProxyType; called per link
spec on start-page render.
4. Manifest M2M subjects — 7 queries via get_m2m_objects loop.
UNION or prefetch collapses.
5. get_publications_preview — full BrowserView pipeline rerun per
preview link spec on start page.
6. lazy_metadata() Comicbox open in v1 stream-link path —
synchronous file I/O on the request thread for partially-imported
books.
7. Progression PUT conflict pre-check — two queries per PUT,
foldable into one conditional UPDATE.
8. Story arcs N+1 in manifest — .only("story_arc", "number") defers
FK; per-row StoryArc.objects.get. Replace with select_related or
.values().
Plan only. No source files touched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* ignore tasks for prettier
* OPDS views perf: Stage 0 — baseline harness + Phase B low-risk wins (#592)
Closes Phase A (R1 OPDS_TIMEOUT=0 rationale, R2 perf harness) and four
of five Phase B items from `tasks/opds-views-perf/99-summary.md`:
- #3 Fix story-arc N+1 in `_publication_belongs_to_story_arcs` —
swap `.only("story_arc", "number")` for
`.select_related("story_arc").only("number", "story_arc__name")`
so `story_arc.name` access doesn't fire one query per row.
- #6 Convert `OPDS2PublicationBaseView.is_allowed` from `@staticmethod`
to instance method reading `self.admin_flags.get("folder_view")`,
and the parallel `OPDS1FacetsView._facet_group` anti-pattern
(`AdminFlag.objects.get` inside a per-facet loop) — both now use
the request-cached MappingProxyType from `SearchFilterView`.
- #13 Extract `_obj_ts(obj)` helper for the
`floor(datetime.timestamp(obj.updated_at))` expression repeated
at six sites across `v2/feed/publications.py` and `v2/manifest.py`.
- #15 Remove dead expression `max(position - 1, 0)` (no assignment)
at `v2/progression.py:226`.
Phase B #12 (`_update_feed_modified` rescan) deferred — it overlaps
with the preview-pipeline pass in Phase D.
The harness lives at `tests/perf/run_opds_baseline.py` with eleven
flows mirroring `tests/perf/run_baseline.py` shape. Cold + warm
captures via django-silk; cold pass invalidates cachalot +
django_cache. Captures `baseline.json` (pre-edits) and
`stage0-after.json` (post-edits) alongside the harness.
See `tasks/opds-views-perf/stage0.md` for the full writeup
including the harness reproducibility note.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 1 — manifest credit + subject batching (#593)
Closes Phase C from `tasks/opds-views-perf/99-summary.md`. Two
manifest items land:
- Tier 1 #2 — `_publication_credits` collapses 11 per-role
`Credit.objects.filter` calls into a single query with
`select_related("person", "role")`. Eliminates the 11-query loop
AND the lazy `credit.person` FK fan-out triggered by
`_add_tag_link` (7 queries on the dev DB's busiest comic). The
role-set partition runs in Python.
- Tier 2 #5 — `_publication_subject` collapses 7 per-model M2M
queries into a single `UNION ALL` over `(pk, name, _kind)` tuples.
Reconstructs `SimpleNamespace` rows so `_add_tag_link` and the
downstream `OPDS2SubjectSerializer` continue to work.
Drive-by: drop the now-unused `get_credits` helper from
`codex/views/opds/metadata.py` (no remaining callers).
`v2_manifest`: 47 → 24 cold queries (-23, ~49%), 113 → 87 ms cold.
Captured `stage1-before.json` + `stage1-after.json` alongside
`stage1.md` for the writeup including the surfaced (but not fixed
here) `peniciller` typo in `OPDS2PublicationMetadataSerializer`.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* speling
* OPDS views perf: Stage 2 — re-enable route caching (#595)
Closes Tier 1 #1 from `tasks/opds-views-perf/99-summary.md` — the
highest-impact item in the entire plan.
- `OPDS_TIMEOUT` flips from 0 (cache_page no-op) to 60 s. Long
enough to amortize a full feed pipeline run across a tab refresh
/ reader-app re-fetch, short enough that bookmark-position
changes show up before the next poll. The disable rationale is
reconstructed in stage0.md § R1.
- New `codex/urls/opds/__init__.py:opds_cached` helper composes
`cache_page(OPDS_TIMEOUT)` with
`vary_on_headers("Cookie", "Authorization")` so the cache key
scopes per-user / per-auth-scheme. Mirrors the binary cover-route
shape at `codex/urls/opds/binary.py`. Applied uniformly across
v1.py, v2.py, and the no-trailing-slash `/opds/v2.0` start in
root.py (which previously bypassed `cache_page` entirely).
- Progression route (`v2/<group>/<pk>/position`) explicitly NOT
wrapped — a PUT mutates the bookmark, and a GET within the cache
window would return stale position. Multi-device sync is the
worst-case (device A PUTs page 100, device B GETs within 60 s
and resumes at the wrong page). The ~9-query / 14 ms cold cost
is small compared to the freshness cost.
Warm-pass measurements collapse to 0 queries / ~1.5–2.7 ms across
every cacheable route (the cache returns the response without
entering the view layer):
v2_manifest 62 → 2.4 ms warm
v2_start 59 → 1.8 ms warm
v2_root_browse 28 → 2.4 ms warm
v1_root_browse 40 → 1.7 ms warm
v1_series_acquisition 24 → 2.7 ms warm
Cold-pass numbers unchanged — the view runs the full pipeline on a
cache miss. Real-world OPDS traffic is dominated by warm hits.
Cross-user isolation verified manually: User A and User B (different
ACL) hit the same URL and receive different payloads (16 223 B vs
15 217 B); each user's warm response matches their own cold
response; `Vary: Accept, Cookie, Authorization, origin` confirmed
on every response.
Captured `stage2-before.json` + `stage2-after.json` alongside
`stage2.md` for the writeup.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* disable discord notification steps in gha
* OPDS views perf: Stage 3 — preview pipeline cache sharing + book queryset joins (#596)
Closes Tier 1 #4 (preview-pipeline re-runs) and the related #17 /
sub-plan 02 #3 (`select_related` shortfall on the OPDS book
queryset).
- `OPDS2FeedLinksView.get_book_qs` overrides the parent
`BrowserView.get_book_qs` to add `select_related("volume",
"language")`. The base method joins `series` only, with an
explicit comment that "OPDS doesn't need volume" — but
`Comic.get_title(volume=True)` reads `obj.volume.name` /
`obj.volume.number_to` per publication, and `_publication_metadata`
reads `obj.language.name`. Without these joins, every publication
iteration fired one lazy `Volume.objects.get` and one lazy
`Language.objects.get`. v1 already does the same join in
`v1/facets.py:64`; this brings v2 to parity.
- `_get_publications_preview_feed_view` shares `_admin_flags` and
`_cached_visible_library_pks` with the parent view. Both are
request-scoped (depend on user, not on params/kwargs), so it's
safe to skip the per-preview re-fetch. Cuts the visible-library
ACL lookup from 1-per-preview to 1-per-request.
`v2_start`: 53 → 29 cold queries (-24, ~45% reduction). Pre-fix
breakdown: 15 codex_volume (N+1) + 12 codex_library (4 per preview
× 3) dominated. Post-fix: 0 codex_volume + 3 codex_library.
Out-of-scope hotspots still visible in the trace (documented in
stage3.md): per-preview age-rating-with-metadata join (4 queries,
needs the bigger UNION-batch rewrite) and the 9 codex_comic filter
/ annotation queries (preserved as the legitimate per-preview
pipeline cost).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 4 — v1 acquisition M2M batching + progression PUT conflict detection (#597)
Closes Phase E from `tasks/opds-views-perf/99-summary.md`.
#7 — Per-page batching of v1 entry M2M fan-out
==============================================
Three new helpers in `codex/views/opds/metadata.py`:
- `get_credit_people_by_comic` — one query for all credits across
all comics on the page; partitions by comic_id in Python.
- `get_m2m_objects_by_comic` — UNION ALL across the 7
`OPDS_M2M_MODELS` tables, partitioned by `(comic_id, kind)`.
`OPDS1EntryData` gains `authors_by_pk` / `contributors_by_pk` /
`category_groups_by_pk` optional dicts. `OPDS1FeedView._get_entries_section`
populates them when `metadata=True` and `key=="books"`. Per-entry
properties read from the dicts when present, fall back to the legacy
single-comic helpers otherwise (so facet entries / single-comic feeds
still work).
Result: 9 queries per entry × N entries collapses to 3 queries per
page. On the harness's "All Batman" series with `?opdsMetadata=1`
(106 comics), `v1_acquisition_with_metadata` drops from 817 cold
queries / 1585 ms to **20 cold queries / 154 ms** (~40× / ~10×) —
verified in a controlled full-feed-state run.
#8 — Progression PUT atomic conditional UPDATE
==============================================
Two changes:
1. `OPDS2ProgressionSerializer.modified` flips from
`read_only=True` to `required=False`. Previously the field was
silently dropped from PUT validated_data, making the conflict
pre-check at `view.py:207-217` unreachable (zero
progression-related queries fired on PUT today).
2. `OPDS2ProgressionView.put` replaces the dead pre-check (which
was `_get_bookmark_query() + qs.first()` — never executed) with
a single atomic conditional UPDATE keyed on
`updated_at__lte=new_modified`. If the UPDATE matches a row,
write succeeds in one query. If 0 rows match AND a bookmark
exists, the DB has a fresher row → 409. If 0 rows match AND no
bookmark exists, fall through to the existing async
`update_bookmark` path (first-time write).
Behavior change: clients that previously sent stale `modified` and
got silent 200s now correctly receive 409 per the OPDS v2
progression spec. Functional verification:
- PUT no `modified` (no bookmark) → 200 (liberal accept, async create)
- PUT with stale `modified` → 409 (atomic conflict detection)
- PUT with fresh `modified` → 200 (atomic UPDATE)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* bump news for progression bugfix
* OPDS views perf: Stage 5 — Tier 3-4 cleanups (#598)
Closes Phase F from `tasks/opds-views-perf/99-summary.md`. Three
items land; two are intentionally skipped after audit.
#16 — `select_related("parent_folder")` on manifest queryset
`OPDS2ManifestView.get_object` adds the join; eliminates the
lazy `Folder.objects.get` per request when folder_view is on.
v2_manifest cold drops 23 → 22 queries.
#11 — Memoize filters JSON via `self.params["filters"]`
`_subtitle_filters` previously re-parsed `request.GET["filters"]`
inline (urllib.parse.unquote + json.loads). The same JSON is
already parsed by BrowserSettingsFilterInputSerializer; reading
from `self.params` skips the third parse. Sub-ms per request,
but on a hot client refresh cumulative.
#10 — Resolve `opds:bin:page` URL once for `_publication_reading_order`
Replace per-page `self.href()` (which fires `reverse()` each call)
with a single sentinel-page resolution + `str.format` substitution.
Saves N-1 `reverse()` calls per manifest hit. Invisible on the
harness's 1-page comic_pk=10785; visible on high-page-count PDFs.
#12 (audited won't-fix): the rescan is functionally necessary —
preview-group mtimes aren't covered by `_get_group_and_books`'s
mtime; removing the rescan would lose preview mtime tracking.
#18 (audited won't-fix): Stage 4 already provides the cleaner
SimpleNamespace pattern; legacy `_add_url_to_obj` fires only on
cold fallback paths where the cachalot reuse risk doesn't apply
(materialized list, not a queryset reused for caching).
After Stage 5 the OPDS perf project has reached the point where
remaining open items either need production telemetry (R3, #19),
medium-risk correctness verification (#9), or are negligible wins
versus framework cost (#14). Recommending pause until production
data points to a specific path.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Reader views perf plan: methodology + 3 sub-plans + ranked backlog (#599)
Audit of `codex/views/reader/` (~850 LOC across 6 source files)
mirroring the OPDS / browser-views perf plan structure. Identifies
15 ranked items + 5 research questions across three view families:
- Reader view chain (`c/<pk>` GET) — params/arcs/books/reader.py
- Reader settings (`c/settings`, `c/<pk>/settings`) — settings.py
- Reader page binary (`c/<pk>/<page>/page.jpg`) — page.py
Top three findings:
1. Comicbox archive open on every page request (sub-plan 03 #1) —
200-page comic = 200 archive opens per read-through. The biggest
single hotsp…
ajslater
added a commit
that referenced
this pull request
May 5, 2026
* update deps
* v1.10.1 fix opds v2 links
* fix news
* fix gha trigger gate logic
* fix gha yaml bug
* fix gha build and deploy from firing on develop
* fix admin update user view & serializer to allow user updates.
* fix gha yaml syntax error'
* fix runing build on develop branch
* fix gha branch gate logic again
* skip tests on deploy if identical files. change discord colors & messages. move ci contaiiner to compose
* fix polling all libraries when no ids submitted in task
* fix poll all library button
* fix opds v2 manifest series link
* force browser reset on start page of opds v2
* fix dev csp for vite hmr
* working server folder picker with enter activates menu and waits to render so the menu isn't lost in the center of the screen
* clean up server folder code to be smaller and remove cruft
* bump news
* attempt to make granian auto reload not hold on to file handles by adjusting template settings in debug mode
* Squashed commit of the following:
commit c76660006840abb36aa37d7355d5c7e242babebf
Merge: b2b5a011a be94a0ae0
Author: AJ Slater <aj@slater.net>
Date: Sun Apr 5 15:49:11 2026 -0700
Merge branch 'develop' into select-multiple
commit b2b5a011ab207a7307505b092877f789e1a66ab3
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:15:54 2026 -0700
fix card menu hover highlighting
commit 5a98fe23d89be5e11fa46ada927100ccd3e08cda
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 22:13:37 2026 -0700
new design for select many mode. no settings drawer involvement. reconfigure cards
commit 82c612e3f77eec90b02465174b49fc8552871686
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:50:11 2026 -0700
add github config to source include. remove dockerhub config
commit b02bd450f4598300d78433a2d749741872d14894
Author: AJ Slater <aj@slater.net>
Date: Sat Apr 4 01:42:32 2026 -0700
update deps
commit d7bbc875e7936d808259c9ba726127fcc3af905e
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 23:56:12 2026 -0700
select many toolbar
commit 246d5ecda59aaabb83e41db3ad32ec552804b9ae
Author: AJ Slater <aj@slater.net>
Date: Fri Apr 3 22:23:24 2026 -0700
select many feature first attempt
* bump news
* efficiency and bugfixes for gha yaml
* fix container name for gha
* fix test report steps running if tests don't run
* use eslint_d for frontend lint
* make sure to build the ci compose locally
* fix too exuberant volumes mounting for gha compose
* remove working dir entry i'm not sure it's good
* add debug line to gha workflow
* remove debug line
* fix test make order
* stop docker compose when done in gha
* i don't think i need DJGNO_SETTINGS_MODULE th ci thing
* fix down call to own bring down ci task
* fix docker compose down"
* fix test results upload
* fix test step name
* fix gha permissions
* bump news
* update deps
* browser set_params method saves last route and saves params to settings
* bump version to 1.10.4
* modify gha discord notification
* use happy-dom for frontend tests
* docker tag script better than the one in ci/
* update deps
* far saner param initialization for browsers and opds start pages
* update dpes & bump alpha version
* v1.10.5a0 (#551)
* bump news
* cirlcle ci no longer handles pre-release
* remove alpha scripts for circleci"
* fix lint-ci
* fix pre-relase gha
* fix default last route on start pages
* bump version
* fix default params
* alpha2
* regular version 1.10.5
* remove develop circleci builds
* adjust variable name
* bump version to 6 alpha 0
* fix default params for feed_views in opds 2
* debug logging for django crash
* v1.10.6a0 (#553)
* bump news
* ignore ruff qa for debug build
* ignore shellcheck"
* lint
* minor refactors of opds v2 feed
* regular v1.10.6 version
* fix docker-tag-latest cscript
* ignore gh token file
* try to log more request errors
* reconfigure logging to hopefully be more verbose about request errors in production
* update deps and bump to alpha version
* bump news (#555)
* v1.10.7
* bump news
* update devenv
* update devevn and deps
* fix pm script
* move django-check to test category
* workflow build frontend and collect static for prodcution build. fix test upload
* bump version and news 1.10.8
* fix dev-module script
* fix news
* explain news
* fix opds clear search setting
* fix clear search button
* use a registry cache instead of gha cache for the dist-builder
* gha use more env vars for image names. retain python dist for 2 days.
* new quick deploy gha script. update deps & devenv.
* silence watchfiles 5 second timeout debug message
* consolidate null values const
* make scope private
* update devenv
* update deps. migrate to unhead v3
* update deps
* fix creating reader global settings
* fix caching
* rename codex build-dist to codex-ci
* fix image name. make gha steps depend on each other more.
* fix gha syntax errors
* names for gha steps
* use ghcr.io for python-debian base
* update deps
* format dockerfile
* picopt treestamps
* fix custom covers not importing. v1.10.11
* fix custom covers count in admin view
* bump news for custom cover count fix
* update deps
* codex identification in server tag and opds generator tag
* update deps
* force no entries on opds start page
* common opds start page mixin. emtpy group objects on start page
* update deps. typechecking.
* api change q to search
* standardize search param as 'search' instead of 'q' or other variations
* remove errant icecream
* clear settings on backend
* Squashed commit of the following:
Fix clear settings null bug, add global settings clear button
- clearComicSettings was setting book.settings to null, causing
Object.entries() to throw TypeError downstream in getBookSettings
- Add null guard in getBookSettings as defense-in-depth
- Add null guard in isClearDisabled computed
- Add clearGlobalSettings action and clear button to Default Settings panel
- Compare against READER_DEFAULTS to determine if global clear is disabled
* simplify settings class hierarchy
* rename select-many store to browser-select-many
* switch to bun. updated devenv
* add a claude md
* use frozenattrdict to speed up configuration
* fix rename of browserSelectMany store
* auth token help
* fix sort-ignores to make deterministic across shells with different locales
* fix crash on settings not being raw
* another gaurd for getMetadta()
* remove keys from unhead meta headers
* fix unhead description for admin tabs"
* fix overzealous lazy importer
* fix lazyImportEnabled variable in metadata-activator
* fix metadata activator from bad cherry pick
* fix errant quote
* fix typechecking
* update devenv & deps
* bump news
* fix import bug linking folders
* fix possible batching crashes. adjust import variables for throughput
* batch comic updates
* move INTERNAL_IPS setting to general django area
* fix typechecking issue
* update deps
* bump version to v1.10.12
* fix redirect on OPDS alternate view with metadata
* minor change to browser empty page for better first time experience
* allow browsing to comics with any top group in opds
* bring project up to speed with bun and no package-lock.json
* fix browser paginator
* bump news for browser paginaor fix
* fix search combobox clearing
* uppercase book close button
* version v1.10.13. update deps
* fix pdfs not displaying
* fix csp for pdfs
* fix OPDS FK constraint failure when session row is missing (#607)
iOS Panels (and other Basic-Auth OPDS clients) intermittently hit
sqlite3.IntegrityError: FOREIGN KEY constraint failed when settings
or bookmarks were saved. Two interacting bugs caused it:
- Janitor cleanup_sessions used `if not session.get_decoded():` to
detect "corrupt" sessions. get_decoded() returns {} for both real
decode failures and legitimate anonymous sessions with no stored
data — exactly what Basic-Auth OPDS clients produce. The nightly
task was wiping valid session rows. Replaced with a direct
signing.loads() call so only genuine signature/decode failures are
flagged.
- _ensure_session_key returned the cookie's session_key without
verifying the row still exists. With cached_db the session loads
from cache without rechecking, so a stale cookie key would slip
through and cause an FK violation when used as SettingsBrowser /
SettingsReader.session_id. Now we verify existence and flush+save
to cycle the key when the row is gone.
Either fix alone closes the user-visible error; both together also
stop the underlying churn that created the bad state.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* extend session-key validation to bookmark + reader-settings paths (#608)
The same stale-session_key FK-violation pattern existed in two more
places that also write rows whose session FK can be stale:
- BookmarkAuthMixin.get_bookmark_auth_filter — feeds session_id into
Bookmark.objects.bulk_create / bulk_update.
- ReaderSettingsBaseView._get_bookmark_auth_filter — feeds session_id
into SettingsReader.objects.create.
Both used the old `if not session.session_key: save()` pattern that
trusts the cookie. Hoist the validated _ensure_session_key helper
from SettingsBaseView up to AuthMixin so every auth-aware view shares
one implementation, and switch both call sites to it. BookmarkAuthMixin
now extends AuthMixin to inherit the helper. BookmarkFilterMixin is
unchanged — it's read-only (filter Q only) and a missing session
correctly returns no rows.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* format
* bump news
* Stats: fix user_registered_count / auth_group_count always-zero bug (#611)
The /admin/stats endpoint's user_registered_count and
auth_group_count fields have been silently returning 0 since at
least Sep 2024. The Stats tab in the admin UI shows 0 registered
users even on installs with multiple accounts.
Root cause: _add_config tried to rename the per-model count keys
produced by _get_model_counts:
config["user_registered_count"] = config.pop("users_count", 0)
config["auth_group_count"] = config.pop("groups_count", 0)
But _get_model_counts builds keys via
``snakecase(model.__name__) + "_count"``. For Django's
``django.contrib.auth.models.User`` / ``Group`` that produces
``user_count`` and ``group_count`` (singular). The pop()s with the
plural names never matched, so the default ``0`` won every time —
and the actual ``user_count`` / ``group_count`` keys were left
orphaned in the dict, then dropped by the StatsConfigSerializer
which only declares ``user_registered_count`` /
``auth_group_count``.
Fix: pop the right source keys.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* defaults for dockerfile ARGs
* format'
* Frontend correctness: 10 bug fixes from sub-plan 01 (#647)
* Reader page: fix load-progress spinner that never appeared
Two stacked bugs in the same setTimeout:
1. Non-arrow callback lost ``this``. The function ran with the
timer's context, not the component's, so ``this.loaded`` and
the write below were both no-ops.
2. The write targeted ``this.loading``, which has never been a
data field on this component. The template binds the spinner
to ``showProgress`` (line 15: ``v-if="showProgress && !loaded"``).
So even if the arrow had been there from the start, the spinner
still wouldn't have rendered — both bugs had to land at once.
Net: ``LoadingPage`` has been dead code for slow image loads.
Switch to an arrow function and write ``showProgress`` instead
of ``loading``. Stash the timer ID so ``beforeUnmount`` can
clear it; a fast page swap mid-delay would otherwise fire the
write on a torn-down component.
Implements B1 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: fix arc-mtime fallback that itself 500'd
``loadMtimes`` builds an arcs list of ``{ group, pks }`` from
``this.arcs``; if the dict is empty the function previously
fell back to ``arcs.push({ r: "0" })``. The comment noted that
"No arcs is a 500 from the mtime api" — the fallback was added
to dodge that 500 — but the wrong-shape fallback also produced
a 500 because the API expects ``group``/``pks`` keys, not ``r``.
Use the canonical shape so the fallback actually works.
Implements B2 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* iOS PWA download: pass the object URL to revokeObjectURL, not the Blob
``URL.createObjectURL(blob)`` returns a ``blob:...`` URL string;
``URL.revokeObjectURL`` must receive that same string to free the
mapping. The previous code passed ``response.data`` (the Blob
itself), which silently no-op'd and leaked one object URL per
download.
On iOS PWAs this matters more than elsewhere because the leak
accumulates across the user's session and can't be reclaimed
short of reloading the app.
Implements B6 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser API: stop mutating caller's settings in getGroupDownloadURL
``getGroupDownloadURL`` did ``delete settings.show`` on the
caller's object before building the URL. Side-effect: any caller
that re-used the settings dict after the download-URL build saw
its ``show`` key silently vanish. This was probably fine when
the function was first written but it's a footgun now that
settings flow through a Pinia store.
Destructure-and-spread to drop ``show`` without touching the
input.
Implements B8 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Auth store: make logout awaitable + clear state unconditionally
Two changes to the logout action:
1. ``async`` so callers can ``await``. The current call site
(``auth-menu.vue``) fires and forgets, but a future UX pass
that wants to disable the button while logout is in flight
needs the promise.
2. Clear ``this.user`` in ``finally`` rather than only on
success. The user clicked "log out" — UI should reflect the
logged-out state immediately, regardless of whether the
server-side logout endpoint succeeded. Server-side cookies
that survive the network failure will get cleaned up by the
next 401.
Implements B7 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser filter menu: stop double-rendering each filter row
``<v-list>`` was passed both ``:items="vuetifyItems"`` AND a
default-slot ``v-for`` over the same list. Vuetify renders the
items prop into ``v-list-item`` children directly, so every row
was being built twice — once by the prop, once by the manual
``v-for``. Visible to users on filter menus with large choice
lists (genres, characters, etc.); each row appeared duplicated
and the DOM cost doubled.
Drop the prop. Keep the ``v-for`` because it carries the custom
``#append`` slot for ``metronName`` rendering. ``:model-value`` /
``@update:selected`` still drive selection state via each list-
item's ``:value`` prop.
Implements B10 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Admin job-tab: remove API-fetching click handler from expanded panel
The expanded status panel had ``@click="loadAllStatuses"`` on its
container div. Any click inside the panel — including clicks on
child elements that bubbled — refetched the entire status map.
Probably copy-pasted as a "refresh on click" gesture, but it
fired far too often: a user inspecting a long status list would
trigger N API calls just from glancing around.
The status data is pushed through the websocket already
(socket.js fans librarian notifications into the admin store),
so the panel is up-to-date without a manual refresh.
Implements B11 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader store: handle bookmark-write errors instead of silently rejecting
``setRoutesAndBookmarkPage`` awaited ``_setBookmarkPage`` but
didn't catch its errors. On a network blip the promise rejected,
the bookmark didn't persist, and the failure became an unhandled
rejection in the browser console — not visible to the user, not
retried, just lost.
Wrap in try/catch. The local page state stays where it is (the
user is reading forward; the bookmark catches up on the next
write), but the failure is logged so debugging surfaces. A
proper user-visible toast + retry path is broader UX work
tracked in the plan.
Implements B3 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: clear progress timer on unmount
``updateProgress`` chains itself via ``setTimeout`` until the
metadata loads or progress reaches 100. The timer ID was never
stashed, so closing the dialog mid-animation left the chain
running — each tick fired on a torn-down component, writing
``this.progress`` and re-scheduling against now-null refs.
Stash the timer ID and clear it in ``beforeUnmount`` so the
chain stops cleanly when the dialog goes away.
Implements B12 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Reader pager: key dynamic component on identity to force remount
``<component :is="...">`` without ``:key`` lets Vue reuse the
existing instance across an ``is`` change when the components
share enough surface (props, name). For the reader's
vertical/horizontal pager swap that's wrong: scroll listeners
attached by the previous mode persist, the new mode's
``mounted`` runs against stale internal state, and any
abort/teardown logic in ``beforeUnmount`` never fires.
Add ``:key="component.name"`` so the swap is a true unmount +
remount — old listeners go away, new mode starts clean.
Implements B13 of tasks/frontend-perf/01-correctness-bugs.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Metadata dialog: lint cleanup for the B12 fixup
Vue option-order rule: ``beforeUnmount`` belongs above
``methods``. Block-comment style required for the multi-line
explanation in ``updateProgress``. Both surfaced when running
eslint on the prior commit; pure cleanup, no behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* news for cherry pick
* OPDS auth: send WWW-Authenticate + opds-authentication content-type on 401 (#652)
Panels and other strict OPDS clients require the WWW-Authenticate header
(per RFC 7235) to trigger the auth prompt, and the spec calls for
application/opds-authentication+json on the auth document. The exception
handler was already converting 403 to 401 with the auth doc body, but
was bypassing DRF's natural 401 response and dropping both the header
and the proper content type, which Panels read as a forbidden state.
Also fix a stray `from re import DEBUG` in the auth view that always
forced the absolute-URL path.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix typing inheritence with drf perms
* bump version to v1.10.14
* update picopt treestamps
* fix typing import error
* fix vulture ignorelist
* readd the mistakenly removed vulture_ignorelist
* v1.10.15 fix show order bug. update deps
* fix nightly job manual expansion
* fix news version number
* V1.11 performance (#714)
* title for age rating tagged
* rename unrestricted to adult for clarity
* remove circleci code
* update to new devenv scripts in frontend
* fix sort-ignores to make deterministic across shells with different locales
* batch comic updates
* fix possible batching crashes. adjust import variables for throughput
* bump news for v1.11.0
* Add features to readme
* add saved view feature
* Add browser-views perf measurement harness (Stage 0) (#574)
Stand up the measurement infrastructure that gates the rest of the
browser-views performance work:
- Add django-silk 5.5 to the dev group; install + route it to a
separate ``silky`` SQLite DB via a new ``SilkRouter`` so profiler
traces never touch the live DB.
- Wire SilkyMiddleware below ServeStaticMiddleware so it only wraps
the API stack, not static-file responses.
- Expose /silk/ under the existing DEBUG URL block.
- Add ``tests/perf/run_baseline.py``: hits the live dev DB via
``django.test.Client``, runs three cold/warm flow pairs (root
browse, filtered search, series metadata), and writes a JSON
baseline artifact. Cachalot + page-cache are invalidated before
each cold pass so the numbers reflect actual DB work.
- Add ``make perf-baseline`` target.
- Commit the per-view analysis docs and initial baseline capture
under ``tasks/browser-views-perf/`` for reference during the
follow-on cleanup stages.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 1 - startup + annotation + filter micros (#575)
Bundle of small, surgical wins observed on the slimlib baseline:
- Cache `libraries_exist` with a 60s TTL in Django's default cache;
invalidate on Library save/delete via lazy-imported signal handlers.
Drops a redundant `Library.objects.filter(...).exists()` from every
browser response.
- Short-circuit the four `_save_browser_*` code paths in
`codex/views/settings.py` when nothing actually changed. Avoids
gratuitous `.save()` calls that flush cachalot on no-op PATCH writes.
- Dedupe the `add_group_by(qs)` call in
`codex/views/browser/browser.py`: group once in
`_get_common_queryset` (no-op for Comic), drop the second call in
`_get_group_queryset`.
- Memoize `get_max_bookmark_updated_at_aggregate` per
`(model, agg_func, default)` on the view instance — the three
callers (group_mtime, order, bookmark) now share one Aggregate.
- Move the `bmua_is_max` flag off the per-row `Value` annotation and
read it from `self.context["view"]` in the browser serializer.
- `@lru_cache(maxsize=256)` on `_preparse_search_query`: extract to a
pure module-level helper keyed on `(text, path_allowed)`; returns
frozen tokens.
- `@lru_cache(maxsize=512)` on `get_field_query`: cache the Lark-parsed
Q tree, `copy.deepcopy` on return because `_hoist_filters` mutates
`child.negated` downstream.
- Stash the `BaseDatabaseOperations(None)` singleton as `_DB_OPS` in
`search/field/expression.py`; `prep_for_like_query` doesn't use the
connection.
- Pre-filter the field loop in `filters/field.py` to keys actually set
in the request — saves ~20 no-op calls per browser.
- Delete `search/field/optimize.py` (`like_qs_to_regex_q` and friends
were already unused — grep confirms only self-references).
Slimlib cold baseline (3 flows, stage1.json vs Stage 0 baseline.json):
- flow_a_root_browse: 21→18 SQL, 189.6→182.3 ms
- flow_b_filtered_search: 21→17 SQL, 187.4→178.0 ms
- flow_c_series_metadata: 34→31 SQL, 251.1→226.9 ms
Warm paths unchanged (0 SQL, ~2 ms). Full tests + ruff pass.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix type warnings
* fix lint error
* format
* Run silk migrations on silky DB at startup (#576)
Stage 0 wired a second SQLite DB (silky) for django-silk profiling traces
and a DATABASE_ROUTERS entry that routes silk app models there. But
ensure_db_schema only invoked `call_command("migrate")` without a database
arg, which only migrates the default DB. The router then blocks silk
migrations from running on default — so the silky DB was never populated,
and the first request through SilkyMiddleware failed with
"no such table: silk_request".
Mirror the pattern from tests/perf/run_baseline.py: after the default
migrate, also run `migrate silk --database=silky` when the silky DB is
configured. Guarded on `"silky" in connections.databases` so production
(DEBUG=False, no silky DB) is a no-op.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 2 — triple COUNT + page mtime cache (#577)
## PR 2a — Eliminate triple COUNT on the paginate path
Each browse request ran three COUNT queries per section (groups & books):
1. The outer grouped COUNT in `_get_common_queryset`.
2. Paginator's internal COUNT triggered by `paginator.page()`.
3. An explicit `.count()` on the paginated slice.
The outer COUNT is needed (sizing the paginator). The other two are
redundant — the page row count is bounded by `per_page` and derivable
from `end_index - start_index + 1`.
- Shadow `Paginator.count` (a `@cached_property`) with the pre-computed
total to skip Paginator's internal COUNT.
- Derive page row count arithmetically from `Page.start_index()` /
`end_index()`.
- Pass `book_count` through `paginate()` alongside `group_count`; drop
`book_qs.count()` on the opds2 path.
- `_paginate_section` returns `(qs, count)` directly.
Short-circuits on `total_count == 0` (avoids Paginator instantiation on
empty sections) and preserves the EmptyPage warning branch.
## PR 2b — Short-TTL page mtime cache
`BrowserView._get_page_mtime()` calls `get_group_mtime(page_mtime=True)`
on every browse request. The query is a filtered Max aggregate that
cachalot caches — but any write to Comic / Bookmark invalidates it, so
bookmark-heavy usage forces recomputation. Cold-path silk traces show
this aggregate at ~26ms on flow_a — the second-slowest query in the
request.
Add a 5s TTL cache layer gated on page_mtime=True. Key includes user,
model, group, pks, page, and a hash of filter-affecting params
(filters, search, q, order_by, order_reverse). The polling MtimeView
path (no page_mtime) is unaffected, so frontend change-detection stays
live.
## Measurements
tests/perf/run_baseline.py on the slimlib DB. Cold = full cache
invalidation; warm = cachalot populated.
| flow | stage1 cold | stage2 cold |
|-------------------------|------------------------|------------------------|
| flow_a root browse | 18 queries / 182.3 ms | 16 queries / 135.6 ms |
| flow_b filtered search | 17 queries / 178.0 ms | 15 queries / 130.3 ms |
| flow_c series metadata | 31 queries / 226.9 ms | 31 queries / 229.9 ms |
flow_a / flow_b: -2 queries, ~26% cold wall-time reduction. flow_c
unaffected (metadata doesn't traverse paginate). PR 2b's benefit is a
dogpile guard after cachalot invalidation — doesn't show in the harness
(cold = both caches empty; warm = cachalot wins first).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix browser paginator
* typecheck browser paginate
* Browser views perf: Stage 3 — cover fan-out collapse (#578)
* Stage 3 — cover fan-out: per-pk endpoint + pre-resolved cover_pk
Before stage 3, every cover card triggered a full BrowserAnnotateOrderView
pipeline to pick the representative comic pk. A default page does ~100 of
these in parallel. This PR collapses that into one correlated Subquery on
the browse response plus a thin per-pk cover endpoint.
Components
- Option A: annotate_order_aggregates(for_cover=True) drops the JsonGroupArray
+ page_count aggregates from the cover path — unused for picking a pk.
- Option B.1: BrowserView group cards get cover_pk / cover_custom_pk via
correlated Subquery that replicates CoverView.get_group_filter exactly
(direct fk match when dynamic_covers/Volume/Folder; sort_name fuzzy match
otherwise, correlated on _GROUP_BY columns so the same comic set as `ids`
is picked — no JSON parsing, no peer aggregate).
- New endpoints /api/v3/c/<pk>/cover.webp and /api/v3/cc/<pk>/cover.webp
serve already-resolved pks with a cheap single-row ACL probe and the
existing CoverPathMixin / CoverCreateThread pipeline.
- Frontend getCoverSrc prefers the new per-pk URL when cover_pk /
cover_custom_pk is on the card; falls back to the old group+pks URL
otherwise, so OPDS and search-active browses keep working.
- FTS skip: cover_pk annotation is skipped when params["search"] is set.
MATCH inside a correlated subquery re-scans the FTS5 index per outer
row (~900ms on a 100-group page). The old URL path still applies the
search filter per cover — same behavior, parallelized over HTTP.
Perf (slimlib dev DB, Flow D = browse + every card's cover):
Flow Before cold After cold Delta
A root browse 156.9ms / 15 q 180.5ms / 15 q +24ms / +0 q
B search 148.3ms / 16 q 132.4ms / 16 q -16ms / +0 q
C metadata 161.9ms / 31 q 165.0ms / 31 q +3ms / +0 q
D browse+covers 2311ms / 1216q 1161ms / 815 q -50% / -33%
Flow A takes a mild regression for the correlated cover subquery, but Flow
D — the realistic user wall-clock — drops by half and 400 SQL queries.
Subsequent cover fetches drop from ~90ms each (full pipeline) to ~5ms
(disk read + 1 ACL probe).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Stage 3 follow-ups: custom_cover URL, FTS pre-materialization, OPDS thin covers
Address review feedback on the cover fan-out collapse:
- Rename /api/v3/cc/<pk>/cover.webp → /api/v3/custom_cover/<pk>/cover.webp.
Descriptive over terse; matches the view name.
- FTS pre-materialization replaces the FTS-skip fallback. A correlated
MATCH re-scans FTS5 per outer row (~900ms on a 100-group page); the
old path worked around this by skipping cover_pk annotation on search
and sending every cover through the legacy pipeline once. We now
pre-select the FTS match set as a non-correlated sub-SELECT in the
outer cover subquery — SQLite materializes it once and each correlated
cover row lookup becomes an indexed pk filter. Cover_pk is annotated
on search responses, the thin endpoint handles each cover.
- OPDS now emits thin-endpoint cover URLs (v1 and v2):
- New OPDSComicCoverByPkView / OPDSCustomCoverByPkView wrap the browser
thin views with OPDSAuthMixin's Basic Auth.
- New opds:bin:cover_by_pk and opds:bin:custom_cover_by_pk URL names.
- v1 _cover_link picks: cover_custom_pk → custom thin; group=='c' → own
pk thin; cover_pk → thin; else legacy group+pks fallback.
- v2 _thumb always uses the thin endpoint (publications are Comic rows
so obj.pk IS the cover pk).
- OPDS inherits annotate_cover() via BrowserView._get_group_and_books,
so group rows already carry cover_pk / cover_custom_pk.
- tests/perf/run_baseline.py gains a flow_e (search + covers) to measure
the search path end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix search combobox clearing
* format
* Stage 3 follow-ups: folder covers, endpoint rename, offline cover pipeline (#579)
- Fix folder cover_pk picking comics not actually in a folder's subtree
by using the recursive ``folders`` M2M (same relation the browse filter
uses) for the per-card cover subquery instead of the direct
``parent_folder`` FK.
- Delete the legacy thick cover endpoint. Rename cover_by_pk.py to
cover.py and drop the "_by_pk" suffix from view class names
(CoverView, CustomCoverView, OPDSCoverView, OPDSCustomCoverView).
Update URL configs, OPDS wrappers, and the frontend client.
- Move all cover generation off the HTTP path. Add CoverCreateTask and
enqueue it from the importer after bulk_create so new comics get
thumbnails pre-warmed offline. When a cached thumb is missing the
per-pk endpoint enqueues the task and responds 202 Accepted with
Retry-After plus the missing-cover placeholder instead of synthesizing
the WebP inline under a worker thread.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update devenv sort ignores
* Force cover refresh after 202 so the placeholder doesn't stick (#580)
Browsers don't honor Retry-After on <img> elements and happily cache the
placeholder served with the 202 response, so even after the cover thread
finishes writing the real thumb the img src keeps rendering the stale
placeholder bytes.
- Backend sends Cache-Control: no-store on the 202 placeholder so the
response isn't cached at that URL.
- BookCover now probes the cover URL with fetch() on mount and, if it
sees 202, waits Retry-After and bumps a reactive `retry` counter that
becomes a cache-busting query param on coverSrc. v-img re-fetches with
the new URL and gets the real cover once the cover thread is done.
Retries are capped at 5 and aborted on unmount via AbortController.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Harden cover endpoint against concurrent read/write races (#581)
Under a full cold-cache page load, 32 HTTP workers hitting the cover
endpoint at the same time as the cover thread was writing thumbs
returned 500s for a handful of covers. The endpoint used a three-step
exists()/stat()/read_bytes() sequence that could race against an
in-flight write, and save_cover_to_cache wrote directly to the target
path so readers could observe a truncated file mid-write.
- save_cover_to_cache now stages to a ``{name}.{pid}.tmp`` sibling and
renames with Path.replace, so a read either sees the pre-existing
state or the fully written file — never a partial one. Stale temps
are unlinked on failure.
- _get_cover_response collapses to a single read_bytes() that catches
FileNotFoundError and logs OSError, then falls through to the 202
path. No race window between stat and read.
- LIBRARIAN_QUEUE.put and the outer get() methods are wrapped in
try/except that log and return the placeholder, so a surprise
exception is never user-visible as a 500 and always leaves a log line.
- cleanup_orphan_covers runs a dedicated ``*.tmp`` sweep over both cover
roots before the orphan scan, so stale temps from crashed writers
don't accumulate.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix custom cover generation
* relegate watch for changes restart to a en env variable settings
* await and finish watch task
* update devenv
* Fix metadata cover fetching pk=0 after Stage 3 cover fan-out (#583)
Stage 3's follow-up commit (2ff08524) dropped the legacy `group+pks`
fallback from the frontend `getCoverSrc()` helper, leaving it to only
build per-pk URLs from `coverPk` / `coverCustomPk`. The browser card
pipeline annotates those fields via `BrowserAnnotateCoverView`, but the
metadata endpoint was never wired into that annotation, and
`metadata-cover.vue` never passed the cover pks through to `BookCover`.
As a result every metadata dialog tried to load
`/api/v3/c/0/cover.webp` — pk 0 — and fell back to the missing-cover
placeholder.
Wire the annotation + serializer + component end-to-end:
- `MetadataAnnotateView` inherits from `BrowserAnnotateCoverView` so
`annotate_cover()` is available on the metadata queryset.
- `MetadataView.get_object()` calls `annotate_cover(qs)` after
`annotate_card_aggregates(qs)`, mirroring `browser.py`'s ordering.
- `MetadataSerializer` exposes `cover_pk` and `cover_custom_pk` as
`SerializerMethodField`s with the same fallback-to-`obj.pk` semantics
that `BrowserCardSerializer` uses — so Comic metadata (`group=c`)
works without annotation, falling back to its own pk.
- `metadata-cover.vue` forwards `md.coverPk` / `md.coverCustomPk` to
`BookCover`.
Verified on /api/v3/{p,i,s,v,c,f}/*/metadata: `coverPk` now resolves
to a real comic pk on every group, and the comic path falls back to
its own pk. No SQL query delta — `annotate_cover` is a correlated
Subquery that inlines into the outer SELECT.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hints (#582)
* Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hydration hints
Fold the three `_intersection_annotate()` passes (value fields, conflict
shadow fields, related-count fields) into a single batched call that
collapses N distinct-count probes into one `aggregate()` and one
`values_list()`, with unique synthetic keys so annotation groups don't
collide.
Extend `FK_QUERY_OPTIMIZERS` so intersecting FK hydration
(`AgeRating`, `Character`, `Team`) carries `select_related` + `.only()`
hints for the nested fields the metadata serializer actually reads
(`AgeRating.metron`, `Character.identifier.url`). Without this, each
nested access fired a follow-up query per instance.
For the M2M intersection path, short-circuit empty-intersection fields
with `Model.objects.none()` so we skip the optimizer setup (and, for
the Comic self-reference path, pointless prefetch dispatches on already
empty results).
Add `flow_c2_comic_metadata` to the perf baseline harness so the
comic-detail metadata path (the FK + M2M heavy flow) is tracked
alongside the series flow.
Measured: flow_c_series_metadata cold SQL drops from 31 → 28 (-3).
flow_c2_comic_metadata baseline captured at 47 queries for future
stages. Other flows unchanged within noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* WATCH for changes variable depends on debug
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Fix folder + FTS crash and preserve rank-ordered cover selection (#584)
* Fix folder browse + FTS crash on search_score ordering
Browsing folders (or any group) with an FTS search and
``orderBy=search_score`` blew up with::
sqlite3.OperationalError: no such column: codex_comicfts.rank
Root cause: Stage 3's cover fan-out collapse builds a correlated
Comic subquery per group card to pick ``cover_pk``. The follow-up
(#579) replaced the correlated ``comicfts__match`` filter with a
non-correlated ``pk__in fts_sq`` pre-materialization — so the Comic
subquery no longer joins ``codex_comicfts``. But
``annotate_order_aggregates`` still annotated
``search_score=ComicFTSRank()`` inside the subquery, and
``add_order_by`` issued ``ORDER BY "codex_comicfts"."rank" * -1``,
which SQLite can't resolve from the subquery's FROM list.
- ``_annotate_search_scores`` now takes ``for_cover`` and skips the
annotation when set. ``annotate_order_aggregates`` threads the flag
through (matching the existing ``for_cover`` pipeline-trims).
- ``_cover_comic_subquery`` passes ``order_key="sort_name"`` to
``add_order_by`` whenever the user's order is ``search_score`` — the
cover's tie-break isn't user-visible, and ``sort_name`` is a real
indexed column on Comic.
Verified: /api/v3/{r,f,a,p}/0/1?q=iron+man&orderBy=search_score all
return 200 with a real ``coverPk`` (not 0). Lint + 20-test pytest
suite pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Preserve FTS rank ordering in cover subquery
The previous fix short-circuited to ``sort_name`` when ``order_by`` was
``search_score`` so a group card's cover could be picked without
referencing ``codex_comicfts.rank``. That fixed the crash but meant the
cover chosen for each card was the alphabetically-first matching comic,
not the top-ranked FTS match — a UX regression on searched pages.
Restore rank-ordered cover selection by:
* Applying ``fts_q`` (``comicfts__match=...``) directly in the cover
subquery so ``codex_comicfts`` is joined and ``rank`` is populated,
while keeping the ``pk__in`` pre-materialization for cheap filtering.
* Teaching ``ComicFTSRank`` to resolve the query-local alias for
``codex_comicfts`` at compile time. Its literal template worked for
the top-level browse query but emitted an unresolvable column ref in
nested subqueries where Django aliases the join as ``V4``/``U1``.
* Skipping ``.group_by("id")`` in the cover-subquery search_score
annotation. The custom force-group-by compiler emits a literal
``"codex_comic"."id"`` that breaks under nested aliasing, and the
cover subquery is already ``.distinct() … LIMIT 1`` so dedup is
redundant there.
Verified against the dev DB: for each publisher card on a ``batman``
search, the annotated ``coverPk`` now matches the top-ranked comic
returned by a direct ``ComicFTSRank`` ordering within that publisher.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update for claude rules
* Browser views perf: Stage 5a — server-side cache_page on cover endpoints (#585)
Wrap /api/v3/c/<pk>/cover.webp, /api/v3/custom_cover/<pk>/cover.webp,
and their OPDS counterparts in cache_page(COVER_MAX_AGE). Compose with
cache_control + vary_on_cookie (API) or vary_on_headers("Cookie",
"Authorization") (OPDS, which accepts Basic + Bearer + Session auth)
so the Vary header is set before cache_page stores the response — the
cache key is keyed per auth identity, no cross-user leakage.
Also raise Django FileBasedCache MAX_ENTRIES from the default 300 to
10000. Cachalot query results + cache_page entries (browser + cover)
exceed 300 during a single browse-with-covers pageload, triggering the
2/3 random cull that silently evicts just-populated cover entries
before the next request can read them. Without this, Flow D warm only
dropped to 743 (~50% cover-cache hit rate); with it, Flow D warm drops
to 0.
Perf impact (stage5a-after.json vs. stage4-after.json):
Flow D — browse + 100 covers warm 802 → 0 queries
Flow E — search + 46 covers warm 368 → 232 queries
Flow E's residual 232 queries are 29 covers that return 202 Accepted
(cover not yet generated; response has Cache-Control: no-store, which
cache_page correctly skips). The 17 covers that returned 200 were 0
queries each. In production, 202s resolve within seconds as the cover
thread catches up, so steady-state warm on Flow E is also 0.
Cold query counts and Flow A/B/C are unchanged.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Raise RLIMIT_NOFILE on startup to avoid macOS 256 FD cap (#586)
Cold browser sessions loading a full 100-card grid intermittently crash
the dev server with `OSError: [Errno 24] Too many open files`. Diagnosis:
- macOS ships a 256 soft cap for RLIMIT_NOFILE (`ulimit -Sn 256`).
- Each Django request thread keeps a sticky SQLite connection
(`CONN_MAX_AGE=600`).
- SQLite WAL mode opens 3 FDs per connection (main + `-wal` + `-shm`).
- The thin cover endpoint dispatches to a `sync_to_async` threadpool, so
a burst of 100 cold cover requests easily spawns ~100 threads → ~300
SQLite FDs, blowing past the 256 cap before page reads even open.
Bumping the soft limit toward the hard cap (or 8192, whichever is lower)
at process start is non-invasive and matches what production deployments
typically achieve via shell `ulimit`. No-op on Linux (already high) and
on platforms without the `resource` module.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps, format json
* Browser views perf: Stage 5b — annotation gating + m2m-aware distinct (#587)
Three surgical correctness/perf changes that surface the right SQL for the
right target. The Stage 5a work (cover endpoint cache_page) collapsed Flow D
warm to 0; 5b cleans up cold-path waste the harness flows don't directly
exercise.
5.2 — Skip ``updated_ats`` JsonGroupArray outside browser/metadata
``obj.updated_ats`` is consumed only in
``BrowserAggregateSerializerMixin.get_mtime`` (browser + metadata
serializers). OPDS computes its own mtime from the bookmark aggregate;
cover/download paths never read it. Skip the DISTINCT scalar aggregate
for those targets.
5.3 — m2m-aware .distinct() in ``BrowserFilterView.get_filtered_queryset``
New ``comic_filter_uses_m2m`` cached property. Comic queries skip
``.distinct()`` unless a real m2m or m2m-through join is present
(story_arc browse, folder browse on cover/choices/bookmark/download
TARGETs, or any m2m field filter). Non-Comic queries still always
``.distinct()`` because the ACL alone traverses ``comic__``
(one-to-many).
5.5 — Tie ``search_score`` ``group_by("id")`` to the same flag
``annotate/order.py:_annotate_search_scores`` only emits the GROUP BY
when fan-out actually exists. Cover path stays gated by ``for_cover``.
5.4 — Skipped intentionally
``codex/views/auth.py`` already caches the three scalar inputs to
``get_acl_filter`` per request. The remaining cost is composing two Q
objects from those scalars — microseconds, not queries. Wrapping a
cached_property keyed on ``(model, user_id)`` would be cosmetic.
20 standalone cases of ``comic_filter_uses_m2m`` pass (every default-target
group, every m2m-folder TARGET, every BROWSER_FILTER_KEY by category, mixed
filters, ``pks=(0,)`` early-out).
stage5b-after.json: existing perf flows preserved (query counts identical;
wall times within run-to-run noise). The wins are on cold paths the harness
doesn't exercise — browsing a Series's comics, default-TARGET folder browse,
OPDS feeds — which a follow-up should add.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5c — batched choices_available (#588)
Replaces the per-field probe loop in BrowserChoicesAvailableView with a
single batched EXISTS annotate. FK fields keep "any non-null exists"
semantics; m2m fields decompose into (has_rel, has_null) booleans plus a
lazy distinct-count probe for the rare has_rel ∧ ¬has_null corner.
The natural EXISTS(SELECT DISTINCT rel ... LIMIT 1 OFFSET 1) form is
broken on SQLite — EXISTS short-circuits on the first row from the
underlying join, before DISTINCT collapses or OFFSET skips — so the m2m
path uses two cheap booleans + a Python-side cap-at-2 distinct probe.
Perf: flow_f_choices_available cold drops 34 → 11 queries (−68%) and
121 → 53 ms (−56%) on the dev DB. Other flows are noise-level
unchanged. tests/perf/run_baseline.py picks up three new flows
(choices_available, m2m field, FK field) so the changed code path is
visible in the artifact.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* run frontend with bun
* fix syntax error in filter sub menu
* Browser views perf: Stage 5d — has_metadata boolean cast + series-browse perf flow (#589)
Cleanup bundle (5.7) shrinks against post-5c code:
- #26: switch ``has_metadata`` annotation from ``F("metadata_mtime")`` to
``ExpressionWrapper(Q(metadata_mtime__isnull=False), BooleanField())`` in
both ``codex/views/browser/annotate/card.py`` and
``codex/views/reader/books.py``. Matches the consumer serializer's
``BooleanField`` and trims the SELECT projection to one byte per row.
- Harness: add ``flow_a2_series_browse`` so the harness covers the
Comic-queryset / no-m2m-filter path that Stage 5b's distinct + group_by
skip wins on. Headline measurement: 6 cold / 4 warm queries, ~13 ms cold.
Items #18 (already absorbed in 5c), #25 (parent-aware reduction already in
place; trimming further would change ORDER BY semantics), #30 (already
coalesced inside ``BookmarkUpdateMixin.update_bookmarks``), and #31
(``zipstream-ng`` already streams via ``FileResponse``) were re-investigated
and skipped with reasons documented in §10 of ``05-replan.md``.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff (#590)
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: 99-summary status column + OPDS perf plan (#591)
* Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff
Locks in the conditional ``codex_comicfts`` demote inside
``BrowserFilterView.force_inner_joins`` so future perf work that strips
the FTS table from the demote set fails loudly instead of re-introducing
``OperationalError: unable to use function MATCH in the requested
context`` on every FTS-enabled browse.
Tests in ``tests/test_search_fts.py``:
* ``test_left_joined_fts_match_raises_operational_error`` — canonical
failure mode. Uses ``Query.promote_joins`` (the documented inverse of
``demote_joins``) to flip the auto-promoted FTS join back to LEFT
OUTER, then proves SQLite refuses the MATCH.
* ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` —
positive contract. ``Comic.objects.values("comicfts__pk")`` joins the
FTS table LEFT OUTER without a non-null filter (so Django's optimizer
leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to
INNER.
* ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` —
negative contract. Same carrier, ``fts_mode=False`` leaves the join
LEFT OUTER.
* ``test_force_inner_joins_unblocks_match_on_left_joined_query`` —
end-to-end repair. The promoted-LEFT-OUTER carrier funneled through
``force_inner_joins(fts_mode=True)`` returns the matching comic.
Also lands the Stage 5e handoff doc that scopes the deferred R3
serializer audit (``stage5e-handoff-serializer-audit.md``) and updates
``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Browser views perf: backlog status column on 99-summary
Stage 5 final exit criterion. Adds a Status column to every tier table in
tasks/browser-views-perf/99-summary.md so the backlog reads as a closed
ledger rather than an open plan.
Markers:
- Tier 1 (4): all four landed (Stage 2, 3, 4, 5a)
- Tier 2 (6): all six landed (Stage 1 / 5b)
- Tier 3 (7): four landed (Stage 2, 5b, 5c), one skipped (#16
subsumed by GroupACLMixin per-request scalar caches), three open
(#11, #12 absorbed by Stage 4 hints; #15 not on a hot Flow A-H path)
- Tier 4 (14): seven landed (Stage 1 / 5c / 5d), four re-investigated
in Stage 5d and confirmed already done (#25, #27, #30, #31), three
open (#22, #24, #29)
- Tier 5 (3): R1 ✅ Stage 5.9 (FTS demote regression test), R2 ❌
skipped (current code is correct), R3 ⏭️ deferred via the
stage5e-handoff-serializer-audit.md brief
Pure documentation update — no code or test changes. Closes the last
unchecked exit criterion on the Stage 5 plan.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: initial planning pass
Mirrors the structure of tasks/browser-views-perf/ for the OPDS surface.
No code changes — produces a ranked backlog so the same per-stage rigor
can be applied to OPDS once browser-views perf wraps.
Files:
- 00-meta-plan.md methodology, scope, sub-plan layout, exit criteria
- 01-routes-and-cache.md OPDS_TIMEOUT=0 disables cache_page on every feed
- 02-feed-pipeline.md v1 + v2 main feeds, BrowserView reuse, preview reruns
- 03-entry-serialization-v1.md v1 entries, lazy_metadata Comicbox open, M2M fan-out
- 04-publications-v2.md is_allowed static-method bypass of admin_flags cache
- 05-manifest.md credit fan-out 11→1, story_arcs N+1, subjects 7→1
- 06-progression-binary-aux.md PUT conflict pre-check + dead expr; binary inheritance
- 99-summary.md 5-tier ranked backlog, phasing A–F, cross-cutting guidance
Top findings (in landing order):
1. OPDS_TIMEOUT = 0 in codex/urls/const.py — every feed wraps
cache_page(OPDS_TIMEOUT) and gets nothing. Single-line config flip
is the highest-leverage win and gates Phase A.
2. Manifest credit fan-out (v2/manifest.py:194-199) — 11 separate
Credit.objects.filter queries because _MD_CREDIT_MAP is iterated.
Collapsible to one query + Python partition.
3. is_allowed static method (v2/feed/publications.py:35-56) bypasses
the request-cached admin_flags MappingProxyType; called per link
spec on start-page render.
4. Manifest M2M subjects — 7 queries via get_m2m_objects loop.
UNION or prefetch collapses.
5. get_publications_preview — full BrowserView pipeline rerun per
preview link spec on start page.
6. lazy_metadata() Comicbox open in v1 stream-link path —
synchronous file I/O on the request thread for partially-imported
books.
7. Progression PUT conflict pre-check — two queries per PUT,
foldable into one conditional UPDATE.
8. Story arcs N+1 in manifest — .only("story_arc", "number") defers
FK; per-row StoryArc.objects.get. Replace with select_related or
.values().
Plan only. No source files touched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* ignore tasks for prettier
* OPDS views perf: Stage 0 — baseline harness + Phase B low-risk wins (#592)
Closes Phase A (R1 OPDS_TIMEOUT=0 rationale, R2 perf harness) and four
of five Phase B items from `tasks/opds-views-perf/99-summary.md`:
- #3 Fix story-arc N+1 in `_publication_belongs_to_story_arcs` —
swap `.only("story_arc", "number")` for
`.select_related("story_arc").only("number", "story_arc__name")`
so `story_arc.name` access doesn't fire one query per row.
- #6 Convert `OPDS2PublicationBaseView.is_allowed` from `@staticmethod`
to instance method reading `self.admin_flags.get("folder_view")`,
and the parallel `OPDS1FacetsView._facet_group` anti-pattern
(`AdminFlag.objects.get` inside a per-facet loop) — both now use
the request-cached MappingProxyType from `SearchFilterView`.
- #13 Extract `_obj_ts(obj)` helper for the
`floor(datetime.timestamp(obj.updated_at))` expression repeated
at six sites across `v2/feed/publications.py` and `v2/manifest.py`.
- #15 Remove dead expression `max(position - 1, 0)` (no assignment)
at `v2/progression.py:226`.
Phase B #12 (`_update_feed_modified` rescan) deferred — it overlaps
with the preview-pipeline pass in Phase D.
The harness lives at `tests/perf/run_opds_baseline.py` with eleven
flows mirroring `tests/perf/run_baseline.py` shape. Cold + warm
captures via django-silk; cold pass invalidates cachalot +
django_cache. Captures `baseline.json` (pre-edits) and
`stage0-after.json` (post-edits) alongside the harness.
See `tasks/opds-views-perf/stage0.md` for the full writeup
including the harness reproducibility note.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 1 — manifest credit + subject batching (#593)
Closes Phase C from `tasks/opds-views-perf/99-summary.md`. Two
manifest items land:
- Tier 1 #2 — `_publication_credits` collapses 11 per-role
`Credit.objects.filter` calls into a single query with
`select_related("person", "role")`. Eliminates the 11-query loop
AND the lazy `credit.person` FK fan-out triggered by
`_add_tag_link` (7 queries on the dev DB's busiest comic). The
role-set partition runs in Python.
- Tier 2 #5 — `_publication_subject` collapses 7 per-model M2M
queries into a single `UNION ALL` over `(pk, name, _kind)` tuples.
Reconstructs `SimpleNamespace` rows so `_add_tag_link` and the
downstream `OPDS2SubjectSerializer` continue to work.
Drive-by: drop the now-unused `get_credits` helper from
`codex/views/opds/metadata.py` (no remaining callers).
`v2_manifest`: 47 → 24 cold queries (-23, ~49%), 113 → 87 ms cold.
Captured `stage1-before.json` + `stage1-after.json` alongside
`stage1.md` for the writeup including the surfaced (but not fixed
here) `peniciller` typo in `OPDS2PublicationMetadataSerializer`.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* update deps
* speling
* OPDS views perf: Stage 2 — re-enable route caching (#595)
Closes Tier 1 #1 from `tasks/opds-views-perf/99-summary.md` — the
highest-impact item in the entire plan.
- `OPDS_TIMEOUT` flips from 0 (cache_page no-op) to 60 s. Long
enough to amortize a full feed pipeline run across a tab refresh
/ reader-app re-fetch, short enough that bookmark-position
changes show up before the next poll. The disable rationale is
reconstructed in stage0.md § R1.
- New `codex/urls/opds/__init__.py:opds_cached` helper composes
`cache_page(OPDS_TIMEOUT)` with
`vary_on_headers("Cookie", "Authorization")` so the cache key
scopes per-user / per-auth-scheme. Mirrors the binary cover-route
shape at `codex/urls/opds/binary.py`. Applied uniformly across
v1.py, v2.py, and the no-trailing-slash `/opds/v2.0` start in
root.py (which previously bypassed `cache_page` entirely).
- Progression route (`v2/<group>/<pk>/position`) explicitly NOT
wrapped — a PUT mutates the bookmark, and a GET within the cache
window would return stale position. Multi-device sync is the
worst-case (device A PUTs page 100, device B GETs within 60 s
and resumes at the wrong page). The ~9-query / 14 ms cold cost
is small compared to the freshness cost.
Warm-pass measurements collapse to 0 queries / ~1.5–2.7 ms across
every cacheable route (the cache returns the response without
entering the view layer):
v2_manifest 62 → 2.4 ms warm
v2_start 59 → 1.8 ms warm
v2_root_browse 28 → 2.4 ms warm
v1_root_browse 40 → 1.7 ms warm
v1_series_acquisition 24 → 2.7 ms warm
Cold-pass numbers unchanged — the view runs the full pipeline on a
cache miss. Real-world OPDS traffic is dominated by warm hits.
Cross-user isolation verified manually: User A and User B (different
ACL) hit the same URL and receive different payloads (16 223 B vs
15 217 B); each user's warm response matches their own cold
response; `Vary: Accept, Cookie, Authorization, origin` confirmed
on every response.
Captured `stage2-before.json` + `stage2-after.json` alongside
`stage2.md` for the writeup.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* disable discord notification steps in gha
* OPDS views perf: Stage 3 — preview pipeline cache sharing + book queryset joins (#596)
Closes Tier 1 #4 (preview-pipeline re-runs) and the related #17 /
sub-plan 02 #3 (`select_related` shortfall on the OPDS book
queryset).
- `OPDS2FeedLinksView.get_book_qs` overrides the parent
`BrowserView.get_book_qs` to add `select_related("volume",
"language")`. The base method joins `series` only, with an
explicit comment that "OPDS doesn't need volume" — but
`Comic.get_title(volume=True)` reads `obj.volume.name` /
`obj.volume.number_to` per publication, and `_publication_metadata`
reads `obj.language.name`. Without these joins, every publication
iteration fired one lazy `Volume.objects.get` and one lazy
`Language.objects.get`. v1 already does the same join in
`v1/facets.py:64`; this brings v2 to parity.
- `_get_publications_preview_feed_view` shares `_admin_flags` and
`_cached_visible_library_pks` with the parent view. Both are
request-scoped (depend on user, not on params/kwargs), so it's
safe to skip the per-preview re-fetch. Cuts the visible-library
ACL lookup from 1-per-preview to 1-per-request.
`v2_start`: 53 → 29 cold queries (-24, ~45% reduction). Pre-fix
breakdown: 15 codex_volume (N+1) + 12 codex_library (4 per preview
× 3) dominated. Post-fix: 0 codex_volume + 3 codex_library.
Out-of-scope hotspots still visible in the trace (documented in
stage3.md): per-preview age-rating-with-metadata join (4 queries,
needs the bigger UNION-batch rewrite) and the 9 codex_comic filter
/ annotation queries (preserved as the legitimate per-preview
pipeline cost).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* OPDS views perf: Stage 4 — v1 acquisition M2M batching + progression PUT conflict detection (#597)
Closes Phase E from `tasks/opds-views-perf/99-summary.md`.
#7 — Per-page batching of v1 entry M2M fan-out
==============================================
Three new helpers in `codex/views/opds/metadata.py`:
- `get_credit_people_by_comic` — one query for all credits across
all comics on the page; partitions by comic_id in Python.
- `get_m2m_objects_by_comic` — UNION ALL across the 7
`OPDS_M2M_MODELS` tables, partitioned by `(comic_id, kind)`.
`OPDS1EntryData` gains `authors_by_pk` / `contributors_by_pk` /
`category_groups_by_pk` optional dicts. `OPDS1FeedView._get_entries_section`
populates them when `metadata=True` and `key=="books"`. Per-entry
properties read from the dicts when present, fall back to the legacy
single-comic helpers otherwise (so facet entries / single-comic feeds
still work).
Result: 9 queries per entry × N entries collapses to 3 queries per
page. On the harness's "All Batman" series with `?opdsMetadata=1`
(106 comics), `v1_acquisition_with_metadata` drops from 817 cold
queries / 1585 ms to **20 cold queries / 154 ms** (~40× / ~10×) —
verified in a controlled full-feed-state run.
#8 — Progression PUT atomic conditional UPDATE
==============================================
Two changes:
1. `OPDS2ProgressionSerializer.modified` flips from
`read_only=True` to `required=False`. Previously the field was
silently dropped from PUT validated_data, making the conflict
pre-check at `view.py:207-217` unreachable (zero
progression-related queries fired on PUT today).
2. `OPDS2ProgressionView.put` replaces the dead pre-check (which
was `_get_bookmark_query() + qs.first()` — never executed) with
a single atomic conditional UPDATE keyed on
`updated_at__lte=new_modified`. If the UPDATE matches a row,
write succeeds in one query. If 0 rows match AND a bookmark
exists, the DB has a fresher row → 409. If 0 rows match AND no
bookmark exists, fall through to the existing async
`update_bookmark` path (first-time write).
Behavior change: clients that previously sent stale `modified` and
got silent 200s now correctly receive 409 per the OPDS v2
progression spec. Functional verification:
- PUT no `modified` (no bookmark) → 200 (liberal accept, async create)
- PUT with stale `modified` → 409 (atomic conflict detection)
- PUT with fresh `modified` → 200 (atomic UPDATE)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* bump news for progression bugfix
* OPDS views perf: Stage 5 — Tier 3-4 cleanups (#598)
Closes Phase F from `tasks/opds-views-perf/99-summary.md`. Three
items land; two are intentionally skipped after audit.
#16 — `select_related("parent_folder")` on manifest queryset
`OPDS2ManifestView.get_object` adds the join; eliminates the
lazy `Folder.objects.get` per request when folder_view is on.
v2_manifest cold drops 23 → 22 queries.
#11 — Memoize filters JSON via `self.params["filters"]`
`_subtitle_filters` previously re-parsed `request.GET["filters"]`
inline (urllib.parse.unquote + json.loads). The same JSON is
already parsed by BrowserSettingsFilterInputSerializer; reading
from `self.params` skips the third parse. Sub-ms per request,
but on a hot client refresh cumulative.
#10 — Resolve `opds:bin:page` URL once for `_publication_reading_order`
Replace per-page `self.href()` (which fires `reverse()` each call)
with a single sentinel-page resolution + `str.format` substitution.
Saves N-1 `reverse()` calls per manifest hit. Invisible on the
harness's 1-page comic_pk=10785; visible on high-page-count PDFs.
#12 (audited won't-fix): the rescan is functionally necessary —
preview-group mtimes aren't covered by `_get_group_and_books`'s
mtime; removing the rescan would lose preview mtime tracking.
#18 (audited won't-fix): Stage 4 already provides the cleaner
SimpleNamespace pattern; legacy `_add_url_to_obj` fires only on
cold fallback paths where the cachalot reuse risk doesn't apply
(materialized list, not a queryset reused for caching).
After Stage 5 the OPDS perf project has reached the point where
remaining open items either need production telemetry (R3, #19),
medium-risk correctness verification (#9), or are negligible wins
versus framework cost (#14). Recommending pause until production
data points to a specific path.
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Reader views perf plan: methodology + 3 sub-plans + ranked backlog (#599)
Audit of `codex/views/reader/` (~850 LOC across 6 source files)
mirroring the OPDS / browser-views perf plan structure. Identifies
15 ranked items + 5 research questions across three view families:
- Reader view chain (`c/<pk>` GET) — params/arcs/books/reader.py
- Reader settings (`c/settings`, `c/<pk>/settings`) — settings.py
- Reader page binary (`c/<pk>/<page>/page.jpg`) — page.py
Top three findings:
1. Comicbox archive open on every page request (sub-plan 03 #1) —
200-page comic = 200 archive opens per read-through. The biggest
single hotsp…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.