feat: scale the deck schema to Anki-sized decks (#43, all phases) - #49
Merged
Conversation
Every published deck cost one homeserver record per card, and manifest.json
carried an unbounded card index (~73 bytes/card). At Anki proportions that
breaks on every axis: a 20k-card deck meant a ~1.46 MB manifest, 20,001 serial
PUTs to publish, 20,000 GETs to open, and a full 1.46 MB rewrite to edit a
single card. A deck tile could not render "20,000 cards" without downloading
all 20,000 entries.
Cards now live batched in `cards/{n}.json` (CHUNK_SIZE = 100), and the manifest
carries only `card_count` plus a `chunks[{n, count, updated_at}]` table. At 20k
cards that is ~3 KB of manifest, ~201 writes to publish, ~200 reads to open,
and ~63 KB to edit one card.
Study order moves onto the card as a sparse `ord` (stride 1000), so inserts
take a midpoint instead of renumbering every following card — which under
chunking would mean rewriting every chunk. Membership is the union of the
chunks, so there is no separate index to drift.
Two bugs fall out of the restructure rather than being patched around:
- CardRepository.upsert/delete never touched the manifest, so a single-card
edit left `cards[].updated_at` stale forever and a delete left a dangling
index entry that fetchByDeck re-GET'd and logged as unreadable on every
load. Single-card writes now go through DeckRepository.upsertCard/deleteCard,
which own the chunk write and the manifest patch together.
- sync() diffed the manifest against the local card cache and issued homeserver
DELETEs for anything missing, so a stale cache could delete a live card.
Membership now falls out of the chunks it reads.
Locating a card to edit uses the chunk mapping the card cache records, so an
edit reads one chunk rather than scanning all of them — pinned by a test.
The app is test-env only with nothing published, so there is no migration and
SCHEMA_VERSION stays at 1; existing test decks must be wiped.
Refs #43
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rewrites Architecture §8.0 (paths, both JSON schemas, all five sync steps) and specs §11.1 for the chunked layout, and adds Architecture §8.4 recording the scale targets, the numbers chunking buys, and the four things still unmeasured — chief among them the homeserver's per-record ceiling, which is what should really set CHUNK_SIZE. Two sections were already wrong before this change and are corrected rather than carried forward: - §8.3 and §12 #6 said SRS is "in-memory in v1; not synced to Pubky". It has been Pubky-backed since SrsStateDto, which said so in its own KDoc. §8.3 also had no row for decks you follow but don't own. - §8.0's path table omitted `srs/` entirely, so it disagreed with PubkyPaths. Also narrows §14's "the triage queue is the reusable spine" claim: bulk import breaks it, since nobody swipes through a 20k-card Anki export. What sources actually share is the parse → preview → commit spine. §5.4 gains the bulk summary screen, and §13 Q4 (max paste size) is resolved. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
Author
|
Follow-ups filed for the gaps this PR records rather than closes:
Remaining #43 phases follow in their own PRs. |
…s alone DeckEditorViewModel.save() called publish() for every change, which rewrites every chunk plus the manifest. Renaming a 20k-card deck therefore re-uploaded all 20,000 cards to change one string — ~201 requests for a field that lives in a single record. Saves that don't touch the card set now go through updateMetadata(), writing one record. Card adds, removes, edits and reorders still republish, since those do change what the chunks must contain. The check compares against the cards as loaded rather than against a count, so swapping a card's text without changing how many there are still republishes. It deliberately ignores updatedAt, which buildCards restamps on every save and which would otherwise report every save as a change. A deck whose manifest has a card count but no chunk table is treated as changed, so a malformed manifest repairs itself on the next save rather than being patched in place. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rogress Every homeserver operation was a serial loop, so a 20k-card publish was 201 round trips end to end and opening that deck was another 200. Publishing, chunk reads, deck listing and the delete sweep now run up to 8 requests at a time via a shared mapConcurrently helper. Bounded rather than unlimited: the FFI's concurrency characteristics are undocumented, and a homeserver may well rate-limit a client that opens hundreds of connections. Two prerequisites had to land first, since concurrency makes both much hotter: - MutableSessionProvider.value was a plain non-atomic var, read on every authenticated call and written by sign-in/sign-out/revalidation. Now @volatile — without it a thread could keep observing a stale secret and retry forever. - SessionRevalidatorImpl serialized revalidation but did not coalesce it: every waiter re-ran the FFI call in turn. It now checks whether the secret it was waiting on has already been replaced and reuses the refreshed session. This matters much more when an expiry fails eight parallel writes at once. Publish is also now interruptible without leaving wreckage. The manifest is written first with `incomplete: true` to claim the deck, then the chunks, then the manifest again to clear the mark. Previously the manifest went last, so a failure part-way through left orphaned chunk records under a deck root that listByAuthor could not see — the user had no way to reach the deck to delete it. Chunk PUTs are idempotent overwrites, so re-running publish simply rewrites them. publish() gains a progress overload feeding Flow-shaped updates into PublishDeckUiState, so a large import shows a real bar instead of a spinner. The chunk counter is mutex-guarded because chunks now complete out of order. Finally, list() is paginated. It has always accepted cursor/limit and no call site used them, so anything past the server's default page was invisible — survivable for a deck listing, not survivable for delete(), which relies on that sweep to avoid orphaning records. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-behind
SRS was one record per card under /decks/{deckId}/srs/{cardId}.json, so a
studied 20k-card deck was ~40,000 homeserver records on top of the deck's own,
and dueForDeck was 20,000 GETs. It also keyed only on deckId, so two authors
whose decks share an id collided in your srs/ tree — #33 blocker 4, and exactly
the collision that following decks from many authors makes likely.
Review state now lives at /pub/loopky/srs/{authorPubky}/{deckId}/{n}.json:
still on *your* homeserver, but keyed by the deck's author and batched. Moving
it out from under /decks/ is the point, not a side effect — your review state
for someone else's deck was never the owner's data, and the old nesting only
looked right while every deck was your own.
SRS has the opposite access pattern to cards: writes are frequent and per-card,
reads need everything at once. So reads are chunked and concurrent, and writes
buffer in memory, flushing per affected chunk. A 30-card session costs one
chunk write instead of 30 record writes.
The flush cannot live in StudySessionViewModel: viewModelScope is cancelled in
onCleared(), so a flush started as the screen goes away would be killed before
finishing — losing exactly the reviews it was meant to save. The repository
owns an app-scoped CoroutineScope and exposes flushAsync() for that case, and
also flushes every 20 reviews so a crash costs a few cards, not a session.
Two things worth noting in the implementation:
- Each state's chunk is recorded when written or loaded, never recomputed. A
flush that derived the chunk differently from the write that dirtied it would
persist into the wrong record.
- The reader discovers chunks by listing the deck's srs/ directory rather than
deriving the range from the card count. Deriving it silently missed any chunk
placed by the fallback assignment used when a card's deck position isn't
loaded, making those reviews permanently unreadable.
Refs #43, #33
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ow foreign refs Three changes, all needed before media works at Anki proportions or across accounts. MediaRepository.get() and delete() resolved paths against the *session* author rather than the deck's, so a blob on any deck you don't own was unreachable — it looked for the file under your own pubky. That silently blocked following a deck with images (#33 blocker 2). get() now takes the deck's authorPubky; call sites thread it through from the deck being displayed. MediaRefDto gains an absolute `uri`, so a card can reference a blob under another author's deck. This is what makes cloning a media-heavy deck viable: the clone points at the original's blobs instead of re-uploading hundreds of MB up front. rehost() then copies one under the clone's own path and clears the uri, so a clone becomes self-contained opportunistically as blobs are used. Content addressing by sha256 makes that swap invisible — the digest is recomputed from the bytes, so the copy lands exactly where a fresh upload would. Media had no cache at all, so a card's image re-downloaded on every recomposition. Adds a small bounded LRU — deliberately small, since it exists to survive recomposition and a back-swipe, not to hold a deck's media. An Anki deck's audio does not fit in memory, which is exactly why the fetch stays lazy and per-card rather than the bulk prefetch Architecture §8.0 used to describe (and which was never implemented). Refs #43, #33 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… preview specs §13 Q4 asked whether 10k chars / 500 cards was the right cap, citing Anki. Now that storage no longer caps deck size, the answer splits in two. The paste box keeps a modest character cap and a raised 2,000-card cap: the constraint there is swipe-triage and a human reading the text, not storage. File import gets parseBulk(), with file-sized limits, and skips the paste box entirely rather than raising its cap to fit a ~2 MB export. parseBulk is the *same parser*, not a second one. Anki's "Notes in Plain Text" export is tab-separated, which spec §6 rule 3 already handles, and its optional third tags column is dropped exactly as any third column is (§8). So Anki .txt import needs zero new dependencies — unlike .apkg, which needs a KMP zip reader, a SQLite driver and zstd. Truncation past the cap was a silent `.take`: an over-long import lost its tail with nothing in the UI to say so. ImportDraft now carries the dropped count. Live-preview parsing was O(n) over the whole input on every keystroke, which is fine for a 40-line paste and not fine as inputs grow. Keystrokes are now debounced; an explicit separator change still re-parses immediately, since the user is waiting on that one. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…riage specs §14 claimed "the triage queue is the reusable spine; every future import source plugs into it". Bulk import breaks that claim, and this is the code that breaks it: nobody swipes through a 20,000-card Anki export one card at a time. File import gets a summary — N cards parsed, what was skipped, what was deduplicated, what was truncated, a three-card sample, one confirm — and then hands off to the same PublishDeckViewModel commit flow paste uses. The spine every source actually shares is parse → preview → commit, not the queue. The file is read on the Android side and the shared ViewModel takes plain text, so the same summary works unchanged for any future source that can produce text. Reached from a secondary "Or import a file" action under the paste CTA; paste stays the primary flow (spec §1). Everything the parse dropped is stated on screen rather than swallowed, which is the whole reason ImportDraft started carrying a truncation count. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured, not anticipated. Publishing a 1,200-card deck end-to-end against a real homeserver failed partway with `429 Too Many Requests` — issue #43 listed FFI/server concurrency behaviour as unknown #2, and this is the answer. Two changes: - The session-authenticated write helpers now retry on 429 with exponential backoff, bounded so a persistently unavailable homeserver still fails rather than hanging. A 429 is transient and the request well-formed, so surfacing it to the user would be wrong. The three near-identical retry helpers collapse into one that handles both session expiry and rate limiting. - MAX_IN_FLIGHT drops from 8 to 4, so a large publish doesn't spend most of its time in backoff. Belt and braces: the retry is what makes it correct, the lower ceiling is what makes it quick. Also surfaces the incomplete flag on deck detail. The failed publish above left a deck reading "900 due · 1200 cards" — the marker manifest worked exactly as designed, keeping the deck reachable and deletable instead of orphaning nine chunk records, but nothing told the user why it was short. The card count comes from the manifest, so without this the deck looks complete while holding fewer cards than it claims. Verified: the same 1,200-card import now publishes cleanly and reads back correctly after a cold restart. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
§8.4 listed four unknowns. Two now have answers from running a 1,200-card import against a real homeserver rather than reasoning about it: the server rate-limits concurrent writes with 429, and an interrupted publish leaves a recoverable deck rather than orphaned records. The remaining unknowns are narrowed to what is genuinely still untested — notably the per-record size ceiling, which is the one that would invalidate CHUNK_SIZE. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#43 §7 assumed .apkg needed three new dependencies — a KMP zip reader, a SQLite driver and zstd — and sequenced it last for that reason. On Android it needs none of them: java.util.zip and android.database.sqlite are both in the platform. So the expensive part of the estimate turned out to be avoidable, and this ships now rather than waiting. An .apkg is a zip around a SQLite collection whose `notes` table joins each note's fields with the ASCII unit separator. Unpacking it to tab-separated text means it feeds the *same* parser as a "Notes in Plain Text" export and the same summary + commit flow — no second import pipeline. Fields beyond the first two are dropped, matching the rule paste already applies to extra columns (spec §8). Anki's inline HTML is stripped to readable text, with whitespace collapsed so adjacent tags don't leave double spaces. Real rich-text fidelity and the Note→Card split stay in #46. Two things are deliberately not handled, and say so rather than failing oddly: `collection.anki21b` (zstd, Anki 2.1.50+) and iOS, which has no platform zip or SQLite exposed to Kotlin/Native and is not runnable end to end anyway. Both point the user at the plain-text export, which works everywhere today. Verified on device: an 800-note .apkg with HTML and a tags field imports, strips to clean text, and publishes as an 800-card deck. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were open in specs §14 and Architecture §12 #6. .apkg shipped without the three dependencies its estimate assumed, and SRS is now chunked, author-scoped and write-behind — so both are recorded as answers rather than left as speculation for the next reader to re-derive. Refs #43 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jvsena42
changed the title
This was referenced Aug 16, 2026
Closed
Owner
Author
|
Filed #55 for the file-import UI. It's honest about what this PR left as a skeleton — in particular four things wired at one end only (publish progress, suggested title, detected separator, |
This was referenced Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements all seven phases of #43. Twelve atomic commits; each phase verified with tests, lint, and an end-to-end run on device against a real homeserver.
Why
The layout assumed small hand-curated decks: one homeserver record per card, plus a manifest carrying an unbounded card index. Real Anki decks are 2k–50k cards.
manifest.jsonWhat's in it
§1 Chunked cards + slim manifest. Cards batch into
cards/{n}.json(100 each); the manifest carriescard_count+ a chunk table and no card index. Study order moves onto the card as a sparseord, so an insert takes a midpoint instead of renumbering everything after it.§1b Cheap metadata saves. Renaming a deck wrote every chunk. Saves that don't touch the card set now write one record.
§4 Concurrency, resumability, progress. Bounded parallelism; the manifest is written first with
incomplete: trueso an interrupted publish stays reachable and deletable instead of orphaning chunks;Flow-shaped progress;list()finally paginated.§6 + §2 Provenance and SRS. A
sourceblock for clone/import lineage. Review state moves to/pub/loopky/srs/{authorPubky}/{deckId}/{n}.json— chunked, and author-scoped so two authors sharing a deck id stop colliding — with in-memory write-behind.§3 Media. Blobs resolve against the deck's author, absolute
pubky://refs let a clone reference an origin,rehost()copies it locally on first use, plus an LRU so images survive recomposition.§5 Bulk import.
parseBulkwith file-sized limits, a summary screen instead of swipe-triage, debounced live preview, and truncation reported instead of silently swallowed.§7
.apkg— see below.Things that turned out differently than planned
.apkgneeded none of its three estimated dependencies. The estimate assumed a KMP zip reader, a SQLite driver and zstd. Android hasjava.util.zipandandroid.database.sqlitein the platform. So it shipped in this PR rather than waiting. Not handled, and reported rather than failed on: zstdcollection.anki21b, and iOS (no platform zip/SQLite for Kotlin/Native, and iOS isn't runnable end to end yet).429. Fixed with bounded backoff-and-retry, andMAX_IN_FLIGHTlowered to 4. This was only found by running it — no unit test would have.Deck.pubkyUriwas left alone. The plan said to route it throughPubkyPaths; that inverts the layering (domain → data, Architecture §4.1).Bugs fixed structurally, not patched
CardRepository.upsert/deletenever touched the manifest, so single-card edits left it permanently stale and deletes left dangling entries re-fetched on every load.sync()diffed the manifest against the local cache and issued homeserver DELETEs for the difference — a stale cache could delete live cards.MediaRepository.get/deleteresolved against the session author, making media on any deck you don't own unreachable (Split keeping someone else's deck into two features: Follow deck (read-only, receives updates) and Clone deck (copy into your account) #33 blocker 2).publish()orphaned chunk records when a deck shrank.MutableSessionProvider.valuewas non-atomic;SessionRevalidatorImplserialized revalidation without coalescing it. Both fixed before enabling concurrency.Breaking
On-homeserver layout changes with no migration, per the issue — nothing is published.
SCHEMA_VERSIONdeliberately stays1. Existing test decks must be wiped; they decode withcard_count = 0.Verification
.txtimport; 800-note.apkgwith HTML and a tags column; and an interrupted publish confirmed to leave a reachable, deletable deck.Refs #43. Unblocks #33 blockers 2, 3 and 4 (blocker 1 is independent and stays there).
🤖 Generated with Claude Code