v3.0.0 — Corpus Intelligence, Authority Linking & Deep Research

[cite] / OpenContracts v3.0.0

The first stable release of the v3 line. v3 began with v3.0.0.a1 in January 2025 and ran through four betas. Since v3.0.0.b4 alone: ~1,400 merged pull requests, 3,541 files changed, 122 new migrations, ~97 security fixes.

The through-line of v3 is that a corpus is no longer a bucket of documents. It is a queryable, self-describing, cross-linked body of work.

📖 Full release notes · 📋 CHANGELOG [3.0.0] · 🔍 Compare v3.0.0.b4...v3.0.0

---

Highlights

Corpus Intelligence & CAML

Corpora describe themselves. CAML (Corpus Article Markup Layer) turns a corpus home into a written article backed by live data, not a widget dashboard.

• Auto-generated Readme.CAML article + corpus logo on creation; logos regenerable via an agent
• Editorial corpus home — Collection Overview, data story, shareable Artifact posters
• Live embeds: extract grids and the reference graph render inline (with bounded payloads)
• Corpus description consolidated onto one canonical source (the Readme.CAML body) with revision history — GraphQL surface unchanged
• One-click collection-intelligence setup for existing corpora

Reference Enrichment & the Authority System

Citations inside documents are extracted, normalized against a jurisdiction's citation grammar, and resolved to authority documents in other corpora.

• Corpus reference enrichment agent with provenance-carrying CorpusReference rows
• Authority corpora + cross-corpus law linking; federal, state, title-identifier, HTS tariff, and Tier-2a municipal citation grammars
• Authority packs ship out-of-repo — manage.py install_authority_pack, AUTHORITY_PACK_ROOTS for sideloading, --check validation. Packs carry their own providers, citation vocabulary, and SSRF source-host allowlists
• Unified Authority Console at /admin/authority (namespaces, relationships, discovery queue, wanted-authorities queue, runs)
• Agentic web locator + discovery providers driving the frontier, rate-capped at the service layer
• Reference-web graph explorer (governanceGraph query); References side panel badges in-flight work and re-links reactively
• Enrichment runner with live WebSocket job status and provisional persistence so long runs survive interruption

Deep Research

• Durable context — a living plan, a memory store, and resume-after-crash, so a multi-hour run survives a worker restart
• Verified citations — finalize checks that each cited passage actually supports its sentence, with a polarity guard; anachronistic citations are refused
• Finding cards — structured, validated output; reports open with the takeaway, not the methodology
• Runs can target a CorpusGroup and are told the cross-corpus retrieval tool exists
• Full frontend + a chat status tool that reports progress mid-run

Search & Retrieval

• Hybrid search — PostgreSQL full-text fused with pgvector similarity via Reciprocal Rank Fusion
• HNSW indexes on every embedding column; Python-side materialization eliminated; pgvector extension to 0.8.0 with Postgres tuned for vector workloads
• Discover is hybrid across every category, plus a new Documents category and cross-content search
• Global post-retrieval reranker
• Multi-corpus retrieval — CorpusGroup + search_across_corpora, with a user-facing management GUI
• OC_SUBTREE_GROUP relationships materialized at ingestion and treated as first-class vector targets, with block-context augmentation and jump-to-relationship deep links

Ingestion at Scale

• Remote-ingest workers — parsing and enrichment offloaded to off-cluster hosts, with hardware-accelerated auto-detecting parser + embedder images
• Service accounts — corpus-scoped access tokens, hashed at rest, a DRF auth backend, and a Worker Account management page
• REST import endpoints (/api/imports/), a database-backed queue, batch processing, multi-queue routing, pre-computed embedding storage
• Chunked, resumable uploads for large files, end to end
• Bulk import carries annotation-to-annotation relationships, link_url/data metadata that survives re-anchoring, and cross-batch document relationships; deferred annotation import re-anchors producer annotations onto the final PAWLs layer
• Optional Gotenberg pre-parse conversion for non-core formats
• Ingestion Monitor admin dashboard for diagnosing ingestion and import failures

Documents, Formats & Storage

• First-class DOCX via the Docxodus microservice — parser, thumbnailer, and a dedicated DocxAnnotator
• Compact PAWLs v2 (~67% smaller) and compact annotation JSON v2 (~75% smaller)
• Within-document index/TOC + sidebar Index tab
• Clickable native PDF link annotations and OC_URL link annotations
• Document lineage + path history; corpus versions, ?v=N deep links, and a version selector
• OPEN_CONTRACTS_V2 export format; export schema bumped to V3

Runtime LLM Configuration

• Provider credentials and endpoints live in the database — change models or keys without a redeploy; secrets encrypted at rest, never exposed via GraphQL
• BaseLLMProvider as a standard pipeline component with four shipped providers
• Resolution chain: per-agent override → per-corpus preferred_llm → install default, with created_with_llm recorded for audit
• Context guardrails — per-model window lookup, token estimation, conversation compaction with persisted bookmarks, tool-output truncation, and in-run history compaction

MCP & Agents

• Authenticated MCP sessions, a create_thread_message write tool, and interactive sign-in for Claude web/desktop and ChatGPT (OAuth discovery, RFC 9728 metadata, CORS)
• Knowledge tools reworked for low friction: unified passage + block feed, annotation content search, list_relationships, bounded get_document_text slicing
• Rate limiting with identity resolution across WebSocket and MCP traffic
• Agent chat: rich-mention delegation between corpora, an agent memory system, tool badges/popovers, and sub-agent approval propagation

Extracts & Analyzers

• Extract iterations & cell-level diff — re-run as a new iteration, then compareExtracts(a, b) for a cell-level diff, with a dedicated frontend
• Agents can dispatch configured extracts and analyses
• Corpus-scoped analyzers, a PII scanner with an auto-annotate tool, HTS tariff-code analysis
• Prompt-injection fencing on per-column constraint fields
• Action Library — CorpusActionTemplate + five default templates + addTemplateToCorpus

Foundation

• GraphQL migrated from graphene to strawberry, with a golden SDL contract and a parity test that fails on any shape drift
• One canonical authorization API (Manager.user_can / obj.user_can) with two-tier caching, pinned by an authorization-invariant suite
• Service-layer centralization across six phases, enforced twice — a pytest architecture test and a Django system check (opencontracts.E001) that fails startup on any inline permission call
• Scoped admin access — superusers are no longer omniscient over user data by default
• mypy in pre-commit and CI, with the auth, users, notifications, analyzer, shared, agents, badges, worker-uploads, and config.* packages graduated off the baseline; a frontend any-baseline gate
• Per-PR changelog fragments under changelog.d/, so concurrent PRs can no longer conflict on CHANGELOG.md
• Large dead-code sweep, including complete removal of Semantic UI React

Frontend

• v3 surface rebrand to [cite] — brand components, /about, chrome + meta, PWA/OG assets, rebranded social-card worker and discovery endpoints
• Dedicated mobile layout for DocumentKnowledgeBase below 768px, plus targeted mobile UX fixes (annotations feed tap-through, type-to-launch Ask bar, tappable Find results)
• Desktop bottom-edge controls consolidated; initial PDF zoom no longer overflows narrower laptops
• Long-form reading layer for CAML articles; Corpus Chat readability and scroll behavior fixes
• Runtime-configurable corpus categories with a superuser CRUD GUI; markdown-customizable profiles; Creative Commons licensing for corpuses

---

🔒 Security

~97 security-tagged changes. Notable:

• SSRF hardening in safe_fetch_bytes — DNS-pinned connections closing the DNS-rebind TOCTOU window, CGNAT gap closed, IPv4-mapped-IPv6 bypass closed, credentials stripped on cross-host redirects, fail-closed on empty DNS results
• IDOR elimination — an IDOR-safe get_for_user_or_none helper, mutation migration, and existence-oracle removal across relationship / metadata-column / datacell mutations
• ZIP import bounds — every unguarded member read (data.json, meta.csv, relationships.csv, sidecars) bounded through one consolidated implementation, with per-sidecar size limits and schema validation
• Auth0 audit — privilege-escalation and credential-storage hardening, is_superuser claim-sync allowlist, bounded JWKS stale-cache fallback, Analysis.callback_token hashed at rest
• WebSocket auth tokens no longer travel in URL query strings
• Secret redaction from logs and mutation inputs; slug-only user privacy across GraphQL; structural annotations no longer leak private documents through a shared structural set
• Outstanding Dependabot / yarn audit alerts resolved; Django container base image bumped

---

⚠️ Breaking Changes

Change	What to do
tokensJsons / boundingBox removed from GraphQL	Use the json field; read it with iter_page_annotations() (Python) or iterPageAnnotations() (TS)
objectSharedWith payload shape changed	Update consumers to the new shape
UserType.canImportCorpus is now nullable	Handle null
pre_authorized_tools no longer controls tool availability	It now governs approval only
Structural annotations are corpus-isolated (since b4)	See the v3 upgrade guide
Export schema bumped to V3	Old exports still import; new exports carry the V3 shape
Semantic UI React removed	Forks importing semantic-ui-react must migrate to @os-legal/ui

---

⬆️ Upgrading

Run migrations first — this release adds 122 migrations since v3.0.0.b4.

docker compose -f production.yml --profile migrate up migrate
docker compose -f production.yml up

Then:

• pgvector 0.8.0 is expected by the vector indexes. The bundled Postgres image ships it; self-managed databases must be upgraded before the HNSW index migrations run.
• Authority packs are no longer in-repo. Install them explicitly with manage.py install_authority_pack <pack>, or point AUTHORITY_PACK_ROOTS at a sideloaded pack directory.
• LLM configuration moved to the database. Env-var config still boots, but the System Settings LLM registry is the runtime authority — set provider credentials there after upgrading.
• DOCX support requires the Docxodus service; Gotenberg is optional for non-core format conversion. Both are in the Compose files.

Full v2 → v3 path: v3 upgrade guide.