Repository navigation
Anki is a two-way trip now: study a deck there, re-import it, and the reviews count. Japanese from the language preset parsed into almost nothing and doesn't any more, a Chinese text no longer arrives as one unclickable word per sentence, and the MECAB magic word is gone from the database.
Studying in Anki now counts
The .apkg exporter has carried LWT's scheduling into Anki since 3.6.0, but the importer discarded everything Anki sent back. Study a deck there, re-import it, and none of it counted.
The importer now reads each card's revlog and replays the grades through LWT's own scheduler. The card's own memory state is deliberately not copied: Anki computes it with its own FSRS parameters, or with SM-2 and no memory state at all, and revlog is the half that records what the learner actually did.
Only reviews later than LWT's own last review are replayed. That is the whole conflict policy and it needs no sync protocol — both sides timestamp their reviews, so the merge is "apply what happened after the state we already hold, in the order it happened" — and it is what stops a re-import applying each review a second time.
Around that:
- "Set due date" and "Forget" come back as themselves. Both write a
revlogrow with no grade on it —ease0, kind Manual — so there is nothing to replay through the scheduler: what the learner said is when, not how well. Those rows used to be dropped, and a card pushed six months out in Anki came back due tomorrow. The due date moves now, no review is invented to explain it, and stability and difficulty are left untouched. - Every card of a note counts. LWT's note type makes one card per note, but a note type edited in Anki can make several, and the reader kept only the first — reviews on the others were silently discarded.
- Notes created in Anki are pointed somewhere. They carry a random guid and no language, so this page cannot create them; it reported a count and left you to work out where your words had gone. It now names Vocabulary → Import an Anki deck, which asks which language they belong to.
Deletions stay impossible, and the page now says so: an .apkg carries no record of them at all — Anki's exporter builds a fresh collection holding only the notes it gathered — so a term missing from a file is left alone rather than guessed at. (#264)
An import that said it had worked, and had not
Anki writes the collection zstd-compressed as collection.anki21b unless Support older Anki versions is switched on, and leaves collection.anki2 behind as a stub holding a single "please update Anki" note.
LWT read that stub quite happily, reported one unrecognised note, and told you the import had succeeded.
It now reads the real collection. The compressed file is SQLite at schema 18, where note types live in notetypes/fields tables rather than the col.models blob, so the field lookup follows col.ver instead of assuming schema 11. Decompression needs ext-zstd, which PHP does not bundle; without it the file is refused with a message naming Anki's own export setting, and the import page stops giving that advice on servers where the extension is present.
This was found by running the round trip against real Anki (pylib 26.08.1) rather than against our own reader — which is also why composer test:anki-oracle now exists. It drives an actual Anki collection: import, answer a card, set a due date by hand, re-export, assert what comes back. The bug above passed the entire existing suite, because every other test of this feature has LWT writing and reading, and the two agreed perfectly while the round trip was broken. (#264)
Japanese parsed into almost nothing
MecabParser read MeCab's character-type column with the test inverted, so every word came out a non-word and every 。 came out a word. A text of twelve words parsed to three — and, since 3.7.0 knows how to say so, raised the "almost none of this text could be turned into words" banner.
It affected any language that actually reached that parser, which is one created from the Japanese preset. The built-in tokenizer has always read the column correctly; this now agrees with it. (#288)
A Chinese text could be one unclickable word per sentence
A language written without spaces gives a regex no gaps to match on, so it took the longest run of word characters it could find — the whole sentence. The reader got a text where 我昨天去了北京大学的图书馆看书 was a single term that could not be clicked, looked up or learned.
Reaching the regex parser at all means no tokenizer ran, so such a language now falls back to splitting each character: what every spaceless preset already asks for, and what getOptedInParserFromRow() already said it relied on. The same text on two differently configured Chinese languages now parses identically. Languages whose words really are space-separated are untouched. (#278)
The banner that fired on the wrong texts
The "none of this text could be turned into words" warning had no minimum length, so 123, 2024, 3.14 and ?!... each sent you off to change a language setting that was working correctly. A price list, a date exercise or a numbers drill genuinely contains no words; the parse was right and now says nothing.
The opposite case was broken too: the density test was switched off below 200 characters, so a 122-character Chinese text on a Latin language — five stray clickable tokens in a wall of unclickable characters — got no explanation at all.
Underneath, the reading view, the check-text page and the API shared the rule for when a parse came out empty but not its input: one measured the raw text, two summed token lengths, so the same text could cross the floor on one surface and not another. ParseCoverage is fed the text token by token and does its own counting now. Coverage is judged as the share of a text's letters that ended up inside words, which — unlike word-per-character density — does not move with how spaced-out a script is, and reads the same at any length. (#289)
The MECAB magic word is retired
LgRegexpWordCharacters is the field that should hold a word-characters regex. Historically it could instead hold the literal MECAB, saying two unrelated things at once — tokenize with MeCab, and this language has no spaces between words. LgParserType says the first properly, but nothing said the second, so twelve sites spelled the comparison out by hand and three got the normalisation wrong.
A migration gives the two jobs to the fields that mean them — LgParserType = 'mecab' and LgRemoveSpaces = 1 — and puts the Japanese preset's real regex back where it belongs, so a migrated language ends up looking like a freshly created one. Every reader moved to those fields first: clearing the marker without that would have dropped Japanese onto the regex tokenizer, where the text still opens and every word in it is wrong. Verified by parsing the same Japanese text before and after the migration across all three legacy flag combinations, with tokens, sentence split and phonetic reading unchanged.
The marker still works if something writes it again — an API client sending the literal, or a language imported from an older install — but a reader that falls back to it now logs a deprecation, and Server Data lists any language still holding it. That page is what people paste into bug reports, which is the half of a deprecation an install owner can act on.
Two casing bugs went with it. GetPhoneticReading tested !== "mecab" in lowercase, so an install holding the marker in any other casing silently got its input back with no error. ExternalParser had the mirror bug — it compared against uppercase only, so the lowercase mecab the language form writes was taken for a regex and compiled as the character class [mecab], making exactly the letters m, e, c, a and b the language's word characters.
MeCab also has one implementation again: ParserRegistry used to route a language naming mecab to a second tokenizer shelling out to the same binary as the built-in one, and the two disagreed. (#288)
The Anki import page returned a 500
/vocabulary/apkg/import had been down since the 3.6.0 release. The router auto-wires the controller, and one interface in its dependency graph (SchedulerInterface) was never bound, so the container threw before the page began.
Nothing caught it: the unit tests build the controller directly with their own services, which is exactly the step that was failing, and no end-to-end test visited the route. It is bound now, both Anki import pages have end-to-end coverage, and the container no longer turns an optional dependency into a required one — a constructor parameter written as ?Foo $foo = null falls back to its default instead of taking the whole page down.
Both Anki forms were also emitting their CSRF field as csrf_token while CsrfMiddleware only ever looks for _csrf_token. They use FormHelper::csrfField() now, which is the one place that knows the name. (#266)
Nine locales that were shipping English
Three pages reported everything they did by echoing markup straight out of their controllers, with no view file and not a single __() call between them — so every label was hardcoded English for all nine shipped locales.
- Both Anki import pages were built entirely from
echostatements — forms,<select>loops, result tables and all. - The term-import page reported all nine of its outcomes — no language selected, invalid language, nothing to import, no term column, no file, wrong format, import failed, and both successes — as ten hand-written copies of the same notification markup.
The markup now lives in one NotificationHelper, every message resolves through the translator, and the controllers do request parsing and nothing else. A dictionary imported into a language other than the current one also stops redisplaying a mislabelled form: it was rebuilt with its language name from the current-language setting but its id from the submitted form, so it could show one language's name while pointing its import and enrichment URLs at another.
On the frontend, shared/stores/statuses.ts has been the intended single source of truth for the word-status model since 3.2.2, but four components kept their own status → colour mapping beside it and had drifted: status 3 was is-info in the word popover and is-warning in the word list; status 99 was three different things in three places. They all read statusTagClass() now, so the word list's colours for statuses 2, 3 and 99 change to match everywhere else. Their labels were hardcoded English too — "Learning (1)", "Well Known", the reading-progress tooltip — and now resolve the same common.status_* keys PHP does.
The admin statistics chart carried a private palette commented as "matching LWT's existing status styles", which it had stopped doing: those were the old 2.x pastels while the reading view moved to a saturated palette, so the same status was one colour in a text and another in the chart, and reading in Dark got light-mode pastels either way. Colours resolve from the --lwt-status* custom properties the text itself is painted with now, so themes apply. (#266, #238)
Config blobs could break out of their script element
Server values reach a page as a JSON island, because the CSP build of Alpine cannot evaluate inline expressions. Each island used to hand-roll its own script tag, escaping flags and parser — and six passed no escaping flags at all, so a value containing </script> closed the element early and the rest parsed as markup. preferences.php separately hand-spliced a pre-encoded value into a JSON literal, which held only while the controller remembered to encode it.
All 35 islands go through ConfigIsland on the PHP side and readPageConfig() on the TypeScript side now, so the escaping is applied in one place, and two tests pin it: one checks the emitter neutralises a hostile payload, the other that no file outside ConfigIsland emits an island itself.
A malformed blob also no longer kills the page. The starter vocabulary page parsed without a try/catch, so a bad blob threw during init() and left an inert shell; readPageConfig() falls back to the caller's defaults. (#301)
Faster, and one page lighter
Difficulty previews stop refetching the book. Each preview downloaded the whole text from Gutenberg — 5.4 s, essentially all network — on every page load, and the suggestion panels ask for one per book they list. FileCache holds the document for 24 hours, keyed by URL, and the same call takes 0.02 s. Only the document is shared; coverage is still computed per user. (#303)
The annotated text view is rendered by the browser. /text/{id}/display was the last page building content in PHP. It fetches GET /api/v1/texts/{id}/annotation now, and three view files become one — the markup unchanged, down to the class names the show/hide buttons rely on. Two empty ruby elements per annotated text go with it: splitting the stored annotation on newlines yields a blank entry for any trailing newline, and the API was reporting those as terms. Annotation items also carry their romanization, resolved for the whole text in one query rather than one lookup per term. (#302)
Under the floorboards
Globals goes from 505 lines to 353. What remains is the request's database handle and user context — the latter being what QueryBuilder scopes every query by, which the class now says outright. Connection owns the mysqli handle alone, so Connection::reset() genuinely clears it rather than recovering a second copy. Globals::table() is gone: it was the identity function, a leftover from a table prefix that no longer exists, called in 185 places and reading as though it did something. (#300)
Another 800 lines of server-side rendering nothing reaches went too — the word-by-word reading pane renderer superseded by text_renderer.ts, three review views behind an unrouted controller method, and a frameset page header from before the app had no framesets. (#299)
And npm run serve now starts the PHP dev server with PHP_CLI_SERVER_WORKERS set; use it instead of a bare php -S, which serves one request at a time. (#303)
Dependencies
Five majors, each verified against the check that would show its breakage: @types/node 25→26, @eslint/css 1→2, purgecss 4→8 (emits byte-identical CSS), and vitest with @vitest/coverage-v8 4→5.
Two stay where they are. Cypress 16 is a regression rather than an upgrade — 45 of 320 tests fail across 9 specs and the suite takes 37:51 against 2:56 on 15, because cy.screenshot() hangs to its 30 s timeout under Electron 41. TypeScript 7 is blocked by @typescript-eslint/parser, which peers <6.1.0.
Three Python floors in the NLP service admitted known-vulnerable releases and were raised: fastapi to 0.109.1 (PYSEC-2024-38, ReDoS), pydantic to 2.4.0 (GHSA-mr82-8j83-vxmv), and python-multipart to 0.0.31, which clears 18 advisories on the old floor. These are >= floors, so nothing changes at install time — what changes is the oldest version a resolver is allowed to pick.
Upgrading is the usual docker compose pull && docker compose up -d --force-recreate. This release carries a migration that rewrites how languages record their tokenizer, so take a dump first, as always. Full detail in CHANGELOG.md.