v26 The Fable edition

While Anthropic was going back and forth with access to Claude Fable, I was reluctant. Why would I was time on something about to go away? And the few attempts I made to use it to help my Multiple Sclerosis research all hit the guardrails like that one time I thought it was a good idea to sprint across the road and got hit by a car. I was about 5 and I spit a baby tooth, not noticing I had a broken collar bone.

Then I figured, why not focus Fable on the back-end things that I can't do alone? So first, I had it look at the database to find my mistakes and bottlenecks, look for optimisations, and be very aggressive about it. Not just because we run Brain Regeneration on a small box, but because the data is growing every day.

Then the Clinical Trials, then the API, then the search, then the data normalization. This turned into a frenzy of sessions ending at 3 am so I could make the most out of my weekly usage.

What came out of this was richer data around clinical trials that is just begging to be put into a proper dashboard that will bring more transparency to the world of clinical research.

Let me cook...

Short and plain english version of changes

• Clinical trials now carry far more structured detail. Phase, recruitment status, study type, and eligibility (including sex) are cleaned up into consistent, searchable values instead of free text each registry sends. Countries are normalized into a real list (with regions), and trials now know their study sites — city, region, and map coordinates — captured from ClinicalTrials.gov and the EU CTIS.
• Sponsors were added to the database. The same sponsor can be spelled five different ways across registries; we fixed that and added filters, and facets to browse research by who is sponsoring it. (Sponsoring and actual Funding are two different things.)
• The site is much faster. Dozens of slow database queries were fixed
• Smarter search. You can now write real boolean searches — ocrelizumab AND (covid OR vaccine OR booster) NOT paediatric, quoted phrases included — and combine multiple subjects, DOIs, or date ranges in one query.
• More trustworthy article data. Retracted papers are now flagged (via Crossref), PDF links are filled in from Unpaywall, duplicate papers sharing a DOI are merged, and the machine-learning models were retrained and fixed so we always serve the latest, correct relevance score.
• Richer author profiles. Authors pull biography and other details from ORCID, and there's a new co-authors view.

⚠️ Before you upgrade:

• Trial contact details were removed from the API. Six contact_* fields (name, address, email, phone, affiliation of a trial's public contact) are no longer returned. They were personal data that the public API should never expose.
• Four deprecated team-scoped endpoints now return 404. They carried deprecation headers for a while; switch to the ?team_id= query-filter replacements.
• Article DOI is now unique. A migration merges duplicate articles that share a DOI. Back up first; review the merge on a copy if you have a lot of duplicates.

Full technical details

Range: v25 (2026-06-10) → main (2026-07-21). ~90 merged PRs, ~380 commits. Migrations 0064 → 0092.

Clinical trials: normalization & enrichment

The largest theme of the release. Each registry (ClinicalTrials.gov, WHO ICTRP, EU CTIS) sends the same concepts in different vocabularies; this release derives clean canonical fields alongside the raw ones so filtering and stats are consistent.

• Normalized derived fields, each stored next to the original:
  • phase_normalized and recruitment_status_normalized (#765), with /trials/stats/ buckets realigned to them (#766).
  • study_type_normalized (#788).
  • inclusion_gender_normalized — trial sex eligibility (#794, #792 predecessor).
  • countries_by_source + a proper TrialCountry relation and regions_normalized (#769), with a backfill_trial_countries command (#771) and fixes for unmapped country/status values found in prod (#770).
• Trial sites (new TrialSite model). Captures per-site city, state/region, and geocoordinates from CTIS retrieve enrichment (#785) and from ClinicalTrials.gov (#790). Exposed via the API as detail-only nesting plus a flat /trials/sites/ endpoint (#791).
• CTIS enrichment. New public-API feedreader for CTIS trials (#778), plus a retrieve-enrichment pass adding all-countries, recruitment dates, and eligibility (#784).
• Intervention modality. Curated modality grouping for categories (#787, migration 0088).
• Ingestion hygiene. HTML stripped from WHO ICTRP fields at ingest (#793); canonical identifier extraction/normalization for trial matching (#727).
• Filtering/stats. Multi-value country filter backed by an indexed EXISTS (#789); recruiting-first ordering on /trials/ (#792); stats facets added to /trials/stats/ (#777).

Sponsors

• Canonicalization data layer — Sponsor / SponsorAlias models, backfills, admin (#779); punctuation-insensitive alias keys (#781); a SponsorMergeCandidate review queue for fuzzy Tier B/C duplicates (#782); canonicalization columns added to the trials xlsx export (#780).
• API surface — nested sponsor object, filters, facets, and a /sponsors/ endpoint (#783).

Performance

A sustained pass over the endpoints that had grown slow with the dataset.

• Replaced the org-visibility join+DISTINCT with an Exists() subquery — ~946ms → ~15ms per request (#747).
• categories monthly_counts N+1 elimination: 17.5s → 1.7s (#749); category count fan-out that was stalling prod fixed (#768, #756).
• Cached /trials/stats/ and /articles/stats/ endpoints (#751); /articles/stats/ relevant count made subject-strict and list-consistent (#753).
• Fixed runaway CPU on /authors/ pagination COUNT(*) (#741), N+1 on /authors/ (#737), and a slow /authors/?orcid= filter via a trigram GIN index (#772, migration 0081).
• N+1 + DISTINCT-pagination trap fixed in search views (#754); dead members prefetch removed from article views (#764).
• True streaming for CSV bulk exports with a site_id filter (#773); N+1 fixes and safety limits for all_results bulk exports (#742).
• CONN_MAX_AGE database connection reuse (#757); gunicorn log level dropped debug→info in prod (#758); Trials.discovery_date index made model-managed (#759).

Search & filtering

• Boolean search parser for ?search= on articles and trials — OR/AND/negation/phrases (#714, #715), with a fix to stop DRF's SearchFilter breaking boolean queries (#716).
• subjects_any OR filter for articles and trials (#712), plus OR semantics in subject filtering (#713).
• Comma-separated DOI list in ?doi= (#717); date-range filters for /articles/ and /trials/ (#705); sort articles by AI relevance / ml_score (#706).

Articles

• Retraction detection via Crossref (crossref_retraction_check, #699, migrations 0064/0066) with regression tests (#703).
• Unpaywall PDF links — pdf_link on Articles, backfilled (#720), with a resumable log and CSV report (#725, #722), and an admin fieldset entry (#721).
• DOI hygiene — unique-DOI constraint merging duplicates (#746, migration 0075); DOI pipeline extracts DOIs from URLs / PubMed API + backfill (#767); comma-separated ?doi= (above).
• access NULL backfilled to "unknown" and normalized at write time (#762, migrations 0076/0077); ml_score field added (#65); “Add Article by DOI” admin action (#726).

Machine learning

• Off-box training: dataset export command + train_models --dataset-file (#735); TF version-parity documentation (#738).
• Fixes: training-data text prep and per-subject labels (#732); model loading / version resolution / batching in predict_articles (#733); BERT max_len default 128 + architecture restore on load (#734); LSTM vectorizer load + trainer metric keys preserved (#736).
• Serve only the latest prediction per (article, subject, algorithm) so stale scores no longer leak (#748, #755).

Authors

• ORCID Tier-1 enrichment fields (#776, migration 0082); biography stored from ORCID + manual recheck button (#729); merge_authors stores bare ORCID IDs (#775).
• Co-authors endpoint + relevant-article counts (#731).

Ingestion reliability

• Pipeline audit — never-miss ingestion, enrichment backoff, takeaways drain (#740); error handling, queue shuffle, better logs (#743); last_successful_fetch_at on sources (migration 0072).
• Feed article titles cleaned of HTML/whitespace (#739); admin summary excludes already-reviewed articles (#709).

Categories

• Per-category (not global) match configuration (#718, migrations 0067/0068).

Ops, CI & housekeeping

• Site-scoped XML sitemap endpoint, subject/relevancy curated (#774).
• Entrypoint automates migrate + collectstatic on startup (#708); docker-compose simplified (#707); CORS headers driven by the debug env var (#711).
• CI gates build and deploy on the Tests workflow passing (#710).
• Logging instead of print (#696); Ruff empty-catch gate E722/S110/S112 (#692); stop admin/search views swallowing all errors as empty results (#694, #695); Django 6.0 format_html fix in admin (#702).
• TeamSerializer whitelisted instead of __all__ (#761).

⚠️ Breaking changes

• Trial contact_* fields removed from the API (#786): contact_firstname, contact_lastname, contact_address, contact_email, contact_tel, contact_affiliation. Model fields retained for internal/admin use only.
• Deprecated team-scoped routes now 404 (#750): GET /teams/<id>/articles/, .../articles/subject/<id>/, .../articles/category/<slug>/, /teams/<id>/subjects/. Use /articles/?team_id=… / /subjects/?team_id=….
• Article DOI uniqueness (#746, migration 0075): duplicate articles sharing a DOI are merged; the constraint is enforced afterward.
• TeamSerializer now whitelists fields (#761) instead of __all__ — a field previously exposed by accident may disappear.
• API write endpoints hardened with per-scheme quota accounting (#752) — review any automated write clients.

Upgrade

1. Back up the database and confirm the dump restores.
2. Apply migrations 0064 → 0092. Several are data migrations (DOI dedup, country/status backfills, access normalization) — review on a restored copy first if you carry many duplicates.
3. Run the trial-country backfill: python manage.py backfill_trial_countries.
4. Update API clients that read trial contact_* fields or the four removed team-scoped routes.
5. Retrain/redeploy ML models off-box if you serve predictions (TF version parity — see #738); the pipeline now serves only the latest prediction per (article, subject, algorithm).