On-demand simplification: assess-only crawl + per-level preview summaries - #698
Merged
Conversation
…ries Move article simplification off the crawl path (copyright + cost): the crawler now only assesses CEFR level, summarizes, and classifies each article - no full simplified bodies. Full simplification stays on-demand (POST /simplify_article/<id>). Gated by SIMPLIFY_AT_CRAWL (default off); set it to restore legacy all-levels crawl simplification. Feed goes originals-first for everyone (flip show_non_simplified_articles and always_open_externally to all users), so feeds no longer depend on pre-simplified inventory. To keep level-appropriate feed-card summaries without simplified child articles, add per-CEFR-level preview summaries: - article_level_summary + article_level_summary_context tables - ArticleLevelSummary / ...Context models; new ArticleLevelSummary context type threaded through ContextIdentifier + Bookmark reverse lookups - the single crawl LLM call now returns a summary per level below the original; each is tokenized and stored - user_article_summary_info and the feed overlay serve the learner's-level summary (tappable, correct tap-context/highlighting), falling back to the article's own summary Server-side only - the client fetches summary tokens by article id and treats context_identifier as opaque, so no web/iOS changes are needed. Not yet validated on device/prod. Migration must run before deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ArticleLevelSummary.ai_model VARCHAR -> ai_generator_id FK to the first-class AIGenerator entity (captures model_name + prompt_version), matching Article.simplification_ai_generator_id. - Bump assess+summarize max_tokens 800 -> 2000 so multi-level summary output (up to 5 levels for a C2 article) is not truncated. - created_at server_default=now() so it is populated via the ORM too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mircealungu
force-pushed
the
wt/on-demand-simplify
branch
from
August 12, 2026 18:12
4c7a4a9 to
05d640f
Compare
|
ArchLens detected architectural changes in the following views: |
…ay query - Extract _select_provider_and_key / _call_simplification_llm / _raise_if_paywall_or_advertorial / _clean_text / _strip_markdown_from_summary; assess_and_summarize and simplify_article_adaptive_levels now share them (removes ~80 lines of duplicated provider/HTTP/parsing boilerplate). - ArticleLevelSummary.pick_best is the single source of truth for level selection; best_for_user_level and the feed overlay both use it. - Overlay now picks the best level per article via a columns-only query, then loads tokenized_summary JSON for only the chosen row per article. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mircealungu
force-pushed
the
wt/on-demand-simplify
branch
from
August 12, 2026 21:17
86001f1 to
7903935
Compare
Add UNIQUE(bookmark_id, article_level_summary_id) (migration + model UniqueConstraint so the SQLite test DB enforces it too), and rewrite find_or_create to insert inside a SAVEPOINT: if a concurrent request created the same row between our SELECT and INSERT, the constraint fires, we roll back just that insert (not the caller's still-open transaction when commit=False), and return the row the other request created. Also fixes the inherited 'except A or B' bug (which only ever caught NoResultFound) by dropping it for one_or_none + explicit IntegrityError handling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mircealungu
marked this pull request as ready for review
August 13, 2026 06:00
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Moves article simplification off the crawl path (copyright + cost) and keeps learners' feeds fair with per-CEFR-level preview summaries. Full simplification stays on-demand (
POST /simplify_article/<id>).Motivation: the crawl was pre-generating simplified bodies for every level of every article (the recent Anthropic monthly-cap exhaustion was this burn), while most crawled articles are never opened.
Changes (API only — no web/iOS changes needed)
1. Crawl → assess-only. The crawler now only assesses CEFR level, writes an abstractive summary, and classifies (article type + disturbing + paywall/advertorial rejection preserved) — no simplified children. Behind
SIMPLIFY_AT_CRAWL(default off); set it to restore the legacy all-levels behavior.2. Originals-first for everyone.
show_non_simplified_articlesandalways_open_externallyflipped to all users, so feeds no longer depend on pre-simplified inventory (also resolves the empty-feed class of bug for B2+/Danish).3. Per-level tappable preview summaries. So each learner still sees a level-appropriate, tappable summary on feed cards without simplified child articles:
article_level_summary+article_level_summary_contexttables (migration26-08-12), models, and a newArticleLevelSummarycontext type threaded throughContextIdentifier+Bookmarkreverse lookups.user_article_summary_infoand the feed overlay serve the learner's-level summary (correct tap-context / highlighting), falling back toarticle.summary.Why zero client changes: the app fetches summary tokens from
/user_article_summary(now level-aware) and treatscontext_identifieras opaque.Tests
19 passing incl. new
test_article_level_summary.py(level selection + context anchoring/fallback).tools/migrations/26-08-12--add_article_level_summary.sqlbefore deploying the code.Not included (intentional / follow-ups)
interactiveSummarypreload slot (perf only; endpoint path already works).🤖 Generated with Claude Code