Per-level card text: fix the collapse, restore per-level titles, un-break deepseek - #717
Merged
Conversation
ArticleLevelSummary rows exist only for levels BELOW the article's own, so
"highest stored row at or below the learner's level" quietly collapses every
reader from the article's level upwards onto the same row.
In Danish that is almost the whole feed: 366 of 368 recently assessed B1
articles carry only A1 and A2 rows, so A2, B1, B2, C1 and C2 readers were all
served the identical A2 summary and changing CEFR level changed nothing on the
card. An A2 reader on an A2 article got the A1 row rather than the article's
own summary, for the same reason.
pick_best now takes the article's own level and returns None at or above it,
which is what best_for_user_level's docstring already promised — the caller
falls back to Article.summary, itself the LLM summary written at the article's
own level. Callers with no article level to pass keep the old behaviour.
The feed overlay joins Article.cefr_level into its columns-only query rather
than reading metrics.cefr_level off the result dicts: that one is the
*effective* level and can come back compound ("B1/B2").
Before on-demand simplification the card DID show a level-appropriate headline: the overlay took title and summary off the level-matched simplified child article. When the crawl stopped generating children (0d047e3), article_level_summary replaced the summary half and nothing replaced the title half — "we don't produce per-level titles" described the hole rather than justifying it. It matters more than it sounds: the default feed view is Headlines, which renders the title and no summary at all, so with no per-level title the CEFR selector changes nothing a default-view reader can see. The assess+summarize call already emits one section per level, so a title per level is a few dozen extra tokens on a call we already make — no new LLM call. - prompt asks for [LEVEL]_TITLE beside [LEVEL]_SUMMARY, headline-shaped and bound to the same subject as the original (it is what the reader taps). - article_level_summary carries title + tokenized_title, both nullable: every existing row has none, and a title dropped by the language check leaves its summary intact. Both fall back to the article's own title. - ArticleLevelTitleContext is a separate join from ArticleLevelSummaryContext even though both point at the same row — title and summary are different token streams, and one table for both would return the title's bookmarks when highlighting the summary. - the feed overlay and the single-article endpoint serve the level title the same way they already serve the level summary. Language check caveat, covered by a test that states it rather than hides it: MIN_CHARS_TO_JUDGE is 60 and headlines are usually shorter, so a wrong-language title alone is not caught. Small residual risk — one call produces the whole response, so an English title arrives beside English summaries, which are long enough to be judged and do trigger the retry.
The assess prompt's only way to report junk was to answer with the bare token "unfinished" — which also let a model end the entire response with one word. deepseek-chat took that exit on EVERY article: measured on 6 complete Danish articles, 6/6 came back as 10 characters in ~1s and were rejected as paywalled. That is why the crawler has been pinned to --provider anthropic since 2026-08-02. It was read as "deepseek-chat regressed", but it is a prompt shape deepseek cannot follow: with the same articles and the same model, moving the signal into INCOMPLETE_ARTICLE / ADVERTORIAL_CONTENT fields gives 6/6 correctly assessed — and truncated copies of those same six (what a paywalled article actually looks like) are still 6/6 rejected. The signal survives; the escape hatch doesn't. This matters beyond deepseek: with one working provider the Anthropic monthly cap starves all assessment in every language, which is what emptied Danish of summaries from 21 Aug (664 consecutive articles with no cefr_level, no summary and no level summaries). The cron pins anthropic, and _select_provider_and_key only fails over on a MISSING KEY, not on a quota error — so nothing degrades gracefully. Backward compatible on purpose: _raise_if_paywall_or_advertorial still rejects a bare "unfinished"/"advertorial", and an absent field reads as NO. The Anthropic account is capped until Sept 1, so this could NOT be re-validated against Haiku — old-shape answers keep working precisely because that is unverified. Only the assess prompt changes. get_adaptive_simplification_prompt has the same bare-word exit and would fail the same way on deepseek, but it runs on anthropic today and is not what starved the crawl; left alone deliberately.
|
ArchLens detected architectural changes in the following views: |
The comment claimed the article title "names the source the learner saw the word in" — which is exactly backwards for a level title: an A1 learner met the word in the A1 headline, not in the publisher's. The behaviour is right, the justification wasn't: get_source_title answers "which document", and the article title is the stable answer where a level title would drift with the reader's CEFR level. States the trade-off it accepts instead of hiding it.
"article level" reads as the article's CEFR level, or as article-level
granularity. The thing stored is an article's text ADAPTED TO a level, and now
that the same row carries a title as well, "..._summary" is wrong outright.
ContextType.ARTICLE_LEVEL_{SUMMARY,TITLE}
-> LEVEL_ADAPTED_ARTICLE_{SUMMARY,TITLE}
article_level_summary -> level_adapted_article_text
article_level_summary_context -> level_adapted_article_summary_context
article_level_title_context -> level_adapted_article_title_context
article_level_summary_id -> level_adapted_article_text_id
Cheap to do now and expensive later: bookmark_context points at context_type by
id, and UPDATE preserves the id, so the 31 existing rows need no migration at
all. The title table and columns are brand new and still empty, so they are
created under the right names rather than renamed afterwards — the migration
supersedes 26-08-28--add_article_level_title.sql and carries the undo for anyone
who applied that draft.
The one thing that is NOT internal: context_identifier is opaque to the client,
which posts it straight back when a word is translated. An app holding a payload
built before this rename would post the old spellings, so from_dictionary accepts
both the old id key and the two old context-type strings. Deletable once no
client can still hold a pre-rename payload; tests pin the behaviour meanwhile.
Those tests earned their keep immediately: the bulk rename had rewritten the
ContextType string VALUES to "LevelAdaptedArticleText" and "ArticleLevelTitle",
which would have written unmatchable context types into the database.
mircealungu
added a commit
that referenced
this pull request
Aug 28, 2026
80% was derived from healthy days measured under Anthropic + the bare-word prompt, which under-enforced the prompt's own "fewer than 3 paragraphs = likely incomplete" rule: only 2-3% of articles were rejected as paywalled, so coverage sat at 97-98%. DeepSeek with the field-based prompt follows that rule literally. The 29 Aug Danish backfill rejected 118 of 629 (19%) — correctly: the rejected set averages 190 words and 1,043 characters against 598 words and 3,246 characters for the assessed set. That is what a paywall stub looks like, not a false positive. So healthy coverage is now ~81%, and an 80% threshold would have paged on a perfectly good day. 70% keeps a real margin under the new baseline while staying far above every capped day (15-38%), which is what the check exists to catch. An alarm that fires on normal operation is worse than no alarm.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started from "titles and summaries don't change when I change my Danish level". Three separate causes, one per commit.
1. Learners at or above the article's level got the wrong summary
ArticleLevelSummaryrows exist only for levels below the article's own, so "highest stored row at or below the learner's level" silently collapsed every reader from the article's level upwards onto the same row.In Danish that is nearly the whole feed: 366 of 368 recently assessed B1 articles carry only A1 and A2 rows, so A2, B1, B2, C1 and C2 readers were all served the identical A2 summary. An A2 reader on an A2 article got the A1 row rather than the article's own summary.
pick_bestnow takes the article's own level and returnsNoneat or above it — which is whatbest_for_user_level's docstring already promised. The caller falls back toArticle.summary, itself the LLM summary written at the article's own level.2. No per-level titles
Before on-demand simplification the card did show a level-appropriate headline — the overlay took title and summary off the level-matched simplified child. When the crawl stopped generating children (0d047e3),
article_level_summaryreplaced the summary half and nothing replaced the title half. The comment "we don't produce per-level titles" described the hole rather than justifying it.It matters more than it sounds: the default feed view is Headlines, which renders the title and no summary at all — so with no per-level title, the CEFR selector changes nothing a default-view reader can see.
The assess call already emits one section per level, so this is a few dozen extra tokens on a call we already make, not a new call.
article_level_summarygainstitle+tokenized_title, both nullable (existing rows have none; a title dropped by the language check leaves its summary intact — both fall back to the article's own title).ArticleLevelTitleContextis a separate join fromArticleLevelSummaryContexteven though both point at the same row: title and summary are different token streams, and one table for both would return the title's bookmarks when highlighting the summary.article.titleandinteractiveTitle.3. deepseek was never broken — the prompt was
The crawler has been pinned to
--provider anthropicsince 2026-08-02 because deepseek-chat "returned unfinished for 90–100% of articles". That is a prompt shape deepseek cannot follow, not a model regression.The assess prompt's only way to report junk was the bare token
unfinished, which also let the model end the whole response in one word. Measured on 6 complete Danish articles: 6/6 came back as 10 characters in ~1s, rejected as paywalled.Moving the signal into
INCOMPLETE_ARTICLE/ADVERTORIAL_CONTENTfields, same articles, same model:The signal survives; the escape hatch doesn't. Verified end-to-end through the real
assess_and_summarize.This matters beyond deepseek: with one working provider, the Anthropic monthly cap starves assessment in every language — which is what left the 664 most recent Danish articles with no
cefr_level, no summary and no level summaries since 21 Aug. The cron pins anthropic, and_select_provider_and_keyonly fails over on a missing key, not on a quota error.Caveats, stated rather than buried
_raise_if_paywall_or_advertorialstill honours a bareunfinished, and why an absent field reads asNO.get_adaptive_simplification_prompthas the same bare-word exit and would fail the same way on deepseek. Left alone deliberately: it runs on anthropic today and is not what starved the crawl.MIN_CHARS_TO_JUDGEis 60 and headlines are shorter. There is a test that asserts this limit out loud rather than implying titles are checked. Small residual risk: one call produces the whole response, so an English title arrives beside English summaries, which are long enough to be judged and do trigger the retry.Deploying
Order matters — the prompt fix must be live before the crawler is pointed at deepseek.
tools/migrations/26-08-28--add_article_level_title.sql--provider deepseek(the comment justifyinganthropicis now obsolete)tools/backfill_reassess_summaries.py --language da --since 2026-08-21 --apply --provider deepseek— one pass gives the stranded articles levels, summaries, per-level summaries and titlesAfter Sept 1, validate Haiku against the new prompt before switching back rather than assuming it.
462 tests pass.