Planned changes for EdgarTools 6.0

[dgunning]

Performance (measure first, then optimize)

• Establish a profiling baseline for the HTML document pipeline and XBRL parsing (large 10-Ks, statement rendering, import time) — all perf work ships with before/after numbers
• Vectorize pandas hot paths in XBRL facts and statements (replace row-wise .apply()/.iterrows())
• Move full-index parsing fully onto pyarrow
• Complete the BeautifulSoup → lxml migration across ~32 modules. Expected gains based on prior measured ports: 5–15x on XML form parsing (Forms 3/4/5, 13F, 13D/G, Form D), 2–5x on DOM-heavy HTML paths like filing.markdown(), and several-fold memory reduction per parsed document. Parse-dominated batch/local-storage workloads should see roughly 2–5x end-to-end
• Fix known parser pathologies: nested-table reprocessing, unbounded colspan/rowspan memory blowups, SGML parser rewrite (~10x faster, ~2.5x less memory)
• Import-time reduction via lazy module loading (design-first; import edgar is ~1s today)

Breaking changes (the 6.0 window)

• Remove the legacy HTML parser (edgar.files, including ChunkedDocument) — edgar.documents is the single parser going forward
• httpx → httpx2 migration (unblocks newer httpxthrottlecache); exception types re-exported so most user code keeps working
• pandas 3.0 support; drop Python 3.10
• Declare the public API: all, stable import paths, internals made explicitly private; some modules move (single-form parsers into subpackages, markdown modules consolidated)
• Consistent error policy: no more silent None returns — absence raises a typed exception from a unified edgar.exceptions hierarchy
• Dependency diet: drop BeautifulSoup, tabulate, and textdistance; replace GPL-licensed unidecode with anyascii; retire unmaintained nest_asyncio
• Naming consistency cleanups (e.g. cash-flow method names unified across API surfaces)

Correctness

• Section extraction overhaul addressing the recently reported defect clusters: item boundaries overflowing into the next section, anchors matching cross-references/table captions, wrong-item resolution, and empty section maps falling back to raw HTML — plus whitespace-tolerant anchor matching and explicit confidence signaling instead of silently returning low-confidence slices
• Continued XBRL standardization fixes (concept-mapping bugs, 497K fund parsing)

Quality & type checking

• Ship a py.typed marker so downstream type checkers see EdgarTools types
• Eradicate stray print statements from library code; audit ~150 silently-swallowed exceptions
• CI ratchets so lint/typing debt can't regrow

Repo & tooling

• Migrate CI and dev workflow to uv (lockfile-based, much faster setup)
• Reorganize the test suite to mirror the package layout; directory-based test categorization
• Significantly reduce clone size (test cassettes/fixtures are ~2 GB today — moving to LFS or an on-demand fetch)
• Dead code removal

Contributor experience

• AGENTS.md + refreshed CONTRIBUTING.md aimed at AI-assisted contributors
• PR template with a verification checklist (ground-truth values from real filings, cassette provenance)
• Public 6.0 milestone with help wanted issues so external work aligns with the roadmap

Sequencing note: everything non-breaking (performance, section fixes, quality, tooling) ships incrementally in 5.x releases as it's ready; the breaking set lands together in the 6.0 window, with a migration guide covering every break.