Skip to content

v5.56.0

Latest

Choose a tag to compare

@dgunning dgunning released this 02 Sep 19:42
· 7 commits to main since this release
4ab709d

Twenty-seven entries, led by the XBRL dimensional and calculation models. Both were keyed too coarsely — axes and domains by concept when they belong to a role, calculation edges by concept when a concept can roll up into two totals — so each returned one answer where the filing gives several.

Upgrading. Several of these change values you already read, all in the direction of returning more or refusing to guess:

  • xbrl.axes and xbrl.domains keep their shape but are now a union across roles rather than whichever role was parsed last. Use the new xbrl.axes_for_role(role_uri) / xbrl.domains_for_role(role_uri) wherever the role is known — the flat views cannot distinguish a domain's two-way breakdown on the income statement from its five-way one in a revenue note.
  • calculation_linkbase() returns more rows: one per filed relationship rather than one per concept. Coca-Cola's FY2023 10-K goes from 278 rows to its filed 306. Code that assumed one row per concept will now see a concept twice, with a different parent and weight — which is what the filing says.
  • get_revenue(unit=...) and get_gross_profit(unit=...) return None for a unit the company does not report in, where they previously returned a USD figure. If you passed unit='EUR' and got a number, that number was dollars.
  • Footnote text changes for footnotes containing markup: text outside a <div> is no longer dropped and nested text is no longer duplicated. related_fact_ids now hold fact ids rather than locator labels, so get_footnotes_for_fact() resolves on filings where it previously returned nothing.
  • Report.get_dataframe() returns data instead of an empty DataFrame, and Document.chunks() no longer raises ModuleNotFoundError. Both were broken for every filing.
  • CurrentReport.doc now warns; it was the last un-warned public route into edgar.files, which 6.0 deletes. Use .document, or .items / report[item].
pip install --upgrade edgartools

Added

  • Role-scoped access to the XBRL dimensional model. XBRL.axes_for_role(role_uri) and XBRL.domains_for_role(role_uri) return the axes and domains a single extended link role declares, keyed on element ID. Prefer them over xbrl.axes / xbrl.domains wherever the role is known: the same domain routinely carries different members in different roles, so the flat views can only offer the union. Axis and Domain gain role_uri, and XBRL.axes_by_role / XBRL.domains_by_role expose the underlying store.
  • CalculationTree.all_arcs, one entry per filed calculation relationship. A concept may roll up into two totals with a different weight, and often a different sign, under each, which the one-node-per-concept all_nodes map cannot represent. all_nodes is unchanged and still the right thing for lookup and membership; all_arcs is the graph.
  • ElementCatalog.typed_domain_ref and ElementCatalog.substitution_group, read from the element declaration, which is what lets Axis.is_typed_dimension report a dimension the filer declared as typed.

Fixed

  • get_revenue(unit=...) answered a request for one unit with a figure in another. When the unit filter correctly rejected every candidate fact, _get_standardized_concept_value fell through to its component calculation, and that calculation checked the two components against each other — which establishes only that they can be added — before normalising with no target unit at all. Apple's GrossProfit and CostOfGoodsAndServicesSold are both USD, so their sum was returned as the answer to unit='EUR' and to unit='shares' alike: 364,357,000,000 in both cases, and both now None. The default USD path is unchanged and still answers from the revenue fact itself. Strictness now reaches the fallbacks, and a derived figure has to answer the unit that was asked for — an exact match always does, a merely compatible one only where the caller did not pin the unit, which is the rule the direct path already used. get_gross_profit() was leaking the same way for a second reason — its own direct branch never asked for strict matching at all, so a compatible-but-different currency succeeded before the guard was even reached — and both fallbacks now answer by the same rule as the direct path. This was latent until bare concept names were indexed (GH #1202, same release): before that, get_fact('GrossProfit') returned None, the components were never found, and the fallback returned None for the wrong reason. (bead edgartools-885h)

  • Report.get_dataframe() returned an empty DataFrame for every filing. FinancialTableExtractor read table_node._processed, an attribute that exists only on the legacy TableNode, while the module parses with the modern one — so every call raised AttributeError into a bare except that returned an empty frame. Measured before the fix: 642 of 642 tables across 21 filings raised, and 0 of 42 Apple R-files produced data; after, 41 of 42, with Apple's Q2 FY2025 net sales reading the filed $95,359M and $219,659M. Two further defects surfaced once the code could run: a short period-header row was padded on the left, attributing every figure to the wrong period, and AUD without a word boundary matched inside "Unaudited", so an unaudited USD statement reported Australian dollars. The money scale now prefers the currency-qualified phrase, since "shares in Thousands, $ in Millions" was read as thousands. (bead edgartools-yq1l)

  • Document.chunks() raised ModuleNotFoundError on every call. It imported edgar.documents.extractors.chunk_extractor, a module that was never written, so this public retrieval API — token-budgeted chunks with overlap, the shape a RAG pipeline asks for — has been unusable since the parser rewrite, along with the DocumentChunk class that was only reachable through it. The module now exists; overlap >= chunk_size raises ValueError rather than advancing the window by zero and looping forever. (bead edgartools-vwtb)

  • Two taxonomies whose namespace URIs end in the same path segment became one concept. A concept is identified by its expanded name — namespace URI plus local name — and the prefix is only a display choice, but when a namespace was declared on the fact element rather than the instance root the parser fell back to deriving the prefix from the URI's final path segment. {http://example.com/l01/alpha/2024}Collision and {http://example.com/l01/beta/2024}Collision both became 2024:Collision: both facts survived, their technical identity did not, and grouping by concept silently merged concepts from different taxonomies. The prefix a document declares is now read wherever it declares it, and a derived segment already claimed by a different namespace is suffixed rather than shared. Real filings declare their namespaces on the instance root, so nothing changes for them — Apple FY2023, Microsoft FY2024 and Coca-Cola FY2023 return the same 1,640 concept strings over 6,138 facts as before, byte for byte. Separately, Axis.is_typed_dimension and Axis.typed_domain_ref were declared on the model and written by no code path at all, so every dimension read as explicit; xbrldt:typedDomainRef is now carried from the element declaration through the element catalog onto the axis. An axis declared only in an imported taxonomy still reads as explicit, because following xs:import into a remote taxonomy is a separate capability that remains open. (GH #1188, #1234, bead edgartools-0c1q.16)

  • Footnote-to-fact links resolved to nothing, and footnote text was truncated and duplicated at once. A footnoteArc's xlink:from names a link:loc whose href fragment is the fact id, and edgartools stored the locator label instead — so Coca-Cola's FY2011 10-K put six ids on its footnote that name no fact, no fact carried a footnote back, and get_footnotes_for_fact() returned nothing for a fact that has one. This resolved almost everywhere by coincidence, because 18,686 of the 18,699 footnote locators in the fixture corpus label a locator with the fact id verbatim; the 13 that do not are in Coca-Cola FY2011 and JPMorgan FY2012, and both are now fully linked. Footnote text was built from descendant <div>s whenever any existed, so direct-child and sibling text was dropped, and since findall('.//div') returns nested divs alongside their ancestors while itertext() already descends, nested text was emitted twice; a footnote in Golub Capital's Q1 FY2025 10-Q lost its opening sentence and its entire table while repeating one sentence, reading 187 words against 352. Text is now assembled in one ordered walk with a break after each block, so words no longer glue across boundaries. Footnote resources are also matched within their own footnoteLink rather than a document-wide dict, since two extended links may legitimately reuse a label — a collision returned a real footnote belonging to a different fact. Of the 123 footnotes across the three fixtures that exercise this, only the 3 with nested markup change; the rest are byte-identical. (GH #1169, #1190, #1230, beads edgartools-0c1q.15, edgartools-05gk)

  • calculation_linkbase() dropped a concept's second calculation parent, and the weight that came with it. Calculation relationships were stored one node per concept, so a concept rolling up into two totals in the same role kept only whichever edge the traversal reached last — and calculation_linkbase() documents its result as one row per parent-to-child relationship. Multiple parents are ordinary rather than exotic: Coca-Cola's FY2023 10-K returned 278 rows against 306 filed arcs, Apple's 194 against 215 and JPMorgan's 438 against 456, and each returns exactly its filed count now. The sign was the sharper half — Coca-Cola files OtherComprehensiveIncomeAvailableforsaleSecuritiesAdjustmentNetOfTaxPortionAttributableToNoncontrollingInterest into one total at +1 and another at -1, so the single row that survived reported a weight whose sign depended on traversal order. CalculationTree now carries all_arcs, one entry per filed edge, beside the all_nodes map that callers use for lookup and membership. The same error at the tree level is fixed with it: a role split across two calculationLink elements, which is legal XBRL, had its second element replace the whole tree built from the first, so relationships are now accumulated across every element and file before the trees are built. (GH #1184, #1238, bead edgartools-0c1q.14)

  • The dimensional model was not role-scoped, so a domain reused across roles kept only one role's members. Hypercubes were stored per extended link role, but axes and domains sat in dicts keyed on element ID alone with no role on either model, and the members of a domain were assigned unconditionally once per role. Apple's FY2023 10-K splits srt:ProductsAndServicesDomain two ways on the income statement and five ways in the revenue note; a single global object held whichever was parsed last, so one of those two roles was always answered with the other's breakdown. Axes and domains are now kept per role, reachable through xbrl.axes_for_role() and xbrl.domains_for_role(), and the existing xbrl.axes / xbrl.domains keep their shape as a union across roles rather than a truncation. Five further gaps in the same loop close with it: xbrldt:targetRole is followed across extended links, so a hypercube whose dimensions are declared in another role is no longer reported as having none; dimension-default arcs set Axis.default_member_id, without which a fact at the axis default cannot be told from an undimensioned one; Table.closed and Table.context_element are read from the all arc instead of being hardcoded; Domain.parent is written for a member that is itself a parent; and axes and members follow the filed order attribute, which was already being extracted and then discarded. Measured on Exxon's FY2022 10-K: 49 of 49 hypercubes now report the closed="true" they were filed with against 0 before, all 39 axes carry their default member against 0, and 37 domains carry a parent against 0. (GH #1171, #1194, #1219, #1225, #1235, #1236, bead edgartools-0c1q.13)

  • get_statement_facts() returned nothing for concepts that are demonstrably in the statement. A fact's statement membership was read from whichever presentation role the enrichment loop reached first and stored as a single scalar, but membership is a set — us-gaap:NetIncomeLoss is presented in the income statement, the cash flow statement, the statement of equity and more. Apple's FY2024 10-K returned 0 rows for NetIncomeLoss in CashFlowStatement, ComprehensiveIncome, StatementOfEquity, Notes and Disclosures, and 6 in each after the fix; statement row counts rise correspondingly, for instance StatementOfEquity from 25 to 63. The full membership is now accumulated once into an index — strictly less work than the per-fact tree scan it replaces — and by_statement_type() tests membership rather than equality. The scalar statement_type keeps its meaning as the primary statement, and the membership lists stay off the declared DataFrame schema. (GH #1242)

  • get_facts_with_dimensions() removed the dimension columns it selected rows on. The predicate matches a fact because it carries dim_* keys, then to_dataframe()'s projection dropped every one of them because include_dimensions was left falsy, so the caller got the right rows with the information that made them the right rows deleted. Intel's FY2024 10-K returned 1,135 correctly-selected rows and 0 dimension columns; it now returns the same rows with all 56. (GH #1243)

  • pivot_by_dimension() silently returned a table that was not a pivot. The lookup column was built by interpolating the caller's spelling, while the projected column is always normalised, so the natural QName form us-gaap:AwardTypeAxis missed and the method fell through to returning the plain unpivoted frame — a DataFrame of plausible shape that a caller who does not already know the pivoted layout cannot distinguish. All three spellings (dei_LegalEntityAxis, dei:LegalEntityAxis, LegalEntityAxis) now resolve to the same pivot, and an axis no fact carries raises ValidationError naming the available axes instead of returning an empty frame that reads as "no data". Both pivot methods also now keep colliding rows apart by unit where that is what distinguishes them, and report any collision they cannot resolve rather than quietly keeping one row of each. (GH #1223)

  • Unitless facts were classified as numeric, so numeric queries returned document metadata. float(value) ran on every fact unconditionally, so dei:DocumentFiscalYearFocus — a gYear with no unit — carried a numeric_value of 2024.0 and a query for facts valued between 2020 and 2030 returned it alongside real monetary facts. XBRL requires a unitRef on numeric items and forbids one on non-numeric items, which makes the unit an exact test where the element catalog is not reachable. Five to seven facts per filing are reclassified — CIK, fiscal-year focus, area code, auditor firm ID, postal code — with every genuine numeric fact untouched. numeric_value is what consumers read as "this is a numeric fact", so this reached well beyond by_value(). (GH #1220)

  • Inline XBRL facts from parse_html() carried no resolved context or unit. _extract_contexts and _extract_units ran namespace-aware XPath (//xbrli:context) against a tree built by lxml.html.fromstring(), whose elements carry the literal tag "xbrli:context" and no namespace URI, so the lookups matched nothing and every fact's .context and .unit came back None while the raw refs sat on the fact untouched. Matching on the local name instead resolves 180,747 of 180,747 context references and 169,255 of 169,255 unit references across the 60 iXBRL filings in the fixture corpus, where the previous count was zero of each. A <unit> built from xbrli:divide now resolves to USD/shares rather than to plain USD, and dimensional members are read from xbrli:scenario as well as xbrli:segment. (GH #1232)

  • A fact's value was the text before its first child, and nothing else. _get_fact_value read element.text, so descendant markup, tail text and the entire continuedAt chain were dropped and the truncated string looked like a complete value — the mechanism behind four separate reports. Relevant content is now assembled properly: descendants and their tails contribute, an ix:exclude subtree does not while the text following it does, escape="true" serializes child markup instead of flattening it, and continuation chains are followed in order with a cycle guard. ix:footnote and ix:continuation are resources rather than facts and are no longer routed through the fact extractor, which removes 451 blank fake facts from the same corpus. Since continuedAt is how issuers split long narrative disclosures, the truncation fell on exactly the text an LLM reads. (GH #1189, #1237, #1239)

  • Unsupported inline transformations passed human display text through as the fact value. The registry held five entries and matched them on an exact QName, so it missed even ixt:num-dot-decimal — the TR4 spelling that accounts for 68,880 of the 90,000 format attributes in the fixture corpus — and returned "Delaware" where the value is "DE" and "ten years" where it is "P10Y". It now implements Transformation Registry 1–4 and the SEC's ixt-sec extensions keyed on a version-independent name, covering every format those 60 filings use, and a format that cannot be applied is recorded on the fact and logged rather than silently yielding display text. Scale is computed with Decimal: applying it in binary float wrote the error into the lexical value, so a filed 0.7 at scale="-2" became 0.006999999999999999 and one filing reported 8199999999.999999 for $8.2 billion. (GH #1250, #1251)

  • Document.has_xbrl reported False on documents with thousands of extracted facts. The inline pre-process pass stores its facts on metadata.xbrl_data, while Document.xbrl_facts called a separate extractor that scanned the node tree for ix_tag metadata the inline pass never sets — two unconnected pipelines, so the public surface answered from one that was never populated. This is worse than missing data, because a working extraction looked like an absent feature and a caller correctly branched away from it. The property now reads the pass that is populated, and has_xbrl is True for all 60 iXBRL filings in the fixture corpus against zero before. (GH #1233)

  • Every XBRL duration was reported one day short. XBRL 2.1 reads a date-only endDate as the end of that day, so the context runs to 24:00 on it and the final day belongs to the period; reporting_periods[*]['days'] subtracted the two dates and dropped it. A calendar year of 2023-01-01 to 2023-12-31 measured 364, Apple's 53-week FY2023 measured 370 instead of 371, and a duration context covering a single day measured 0. The count now lives in one place, edgar.xbrl.core.duration_days, used for the stored field and for the two sites in period_selector that recompute a duration and compare it against that field. periods.py keeps its own exclusive count, because its bucket bounds were calibrated against it and a 53-week fiscal year sits exactly on the <= 370 bound - it no longer writes that local value back over the public field, which is what made the field mean different things depending on whether a statement had been rendered first. Measured over the 8 filings under data/xbrl/datafiles: 191 duration periods change their days value and nothing else changes - no period_type, fiscal_period, label or key, and no change to the periods selected for any statement. (GH #1247)

  • Table rows were dropped at random. <thead> rows were deduplicated by id(), but nothing kept the lxml element proxies alive — lxml frees a proxy once unreferenced and CPython reuses the address, so a tbody row could inherit a stale thead id, match it, and be skipped as "already processed". The row vanished from the table with no error. Measured on the ABRAMS Form 4 0000001923-04-000001: 1 to 22 divergences per 25 renders before, 0 in 100 after. Membership is now tested by identity against a list held for the whole function, so the collision is impossible by construction rather than dependent on GC timing. This is what made the regression lane fail at random. (bead edgartools-gf6v)

  • EntityFacts lookups reported that present data did not exist. The fact index was keyed by the qualified name a fact is tagged with (us-gaap:StockholdersEquity) and by the lowercased label, never by the bare concept name — so get_annual_fact('StockholdersEquity') returned None and warned "No fact found", while the value was present and correct. get_annual_fact('Assets') worked only because us-gaap:Assets happens to be labelled "Assets" and matched the label key by coincidence, which is what made this look intermittent rather than systematic: in Snowflake's company facts that coincidence holds for 4 of 339 concepts. The local name is now indexed too, in its filed case — a lowercased form would merge with existing label keys. Across AAPL, JPM and KO, 1,954 of 4,264 concept and label lookups that previously answered None now return a fact, with no existing answer changed or lost. get_fact() and available_periods() shared the defect and share the fix. (GH #1202)

  • A filtered or paged Reports collection returned reports whose content could not be read. Report.content fetches its R-file through reports._filing_summary._filing_sgml, and three of the four methods that return a new collection rebuilt it without that back-reference, so filter(), next() and previous() handed back reports with the right filenames and raised AttributeError: 'NoneType' object has no attribute '_filing_sgml' on .content. On Apple's FY2024 10-K a multi-row filter now reads R3.htm as the 79,891 characters get_by_category() always returned. Only the multi-row branch of filter() was affected — filtering to a single report returns a Report still attached to the original collection, which is why this stayed hidden. Deriving a collection now goes through one place, so the next method to return a subset carries the context by default. (GH #1191)

  • FactQuery.scale() was a silent no-op on every parsed fact. Transforms were applied to fact['value'], which holds the filed string, so scale()'s own isinstance(value, (int, float, Decimal)) guard rejected it and handed it straight back — while numeric_value, the float every consumer actually reads, was never transformed on any path. scale(1000), the example in its own docstring, changed nothing. A numeric fact is now transformed on its numeric value and both fields are written together so they cannot disagree; a non-numeric fact keeps being transformed on value, so text transforms still work and scale() leaves TextBlocks alone through the same guard. StitchedFactQuery duplicated the defect and shares the fix. (GH #1187)

  • FactQuery.to_dataframe() served a stale table after the query was narrowed. The cache was keyed on the column projection alone, so the first to_dataframe() answered every later one: after .by_concept('us-gaap:Assets'), execute() returned 2 rows while to_dataframe() still returned the original 1,075, with nothing to say the two disagreed. FactQuery is a fluent mutable builder whose methods return self, so one object describes a different population after each call, and the key now covers the query configuration as well as the projection. Identical queries are still served from the cache. StitchedFactQuery duplicated this one too. (GH #1186)

  • A disclosure whose subject is notes payable was routed to notes(). get_all_statements() falls back to keyword matching when a role carries no FilingSummary menu category, and that fallback tested the bare substring "note" before it tested "disclosure". "Note" names a financial-statement section and a debt instrument, so Oracle's four Disclosure - NOTES PAYABLE AND OTHER BORROWINGS roles (10-Q 0000950170-23-047713) reported type="Notes" / category="note", came back from xbrl.notes(), and were absent from xbrl.disclosures(). The ambiguous word no longer outranks a role that states what it is - by its Disclosure category marker, or by the concept it hangs from (us-gaap_DebtDisclosureAbstract) - unless the definition names the notes section itself, so Notes to Consolidated Financial Statements and Note 1 - Organization still classify as notes. Roles whose definition does not contain "note" are classified exactly as before. (GH #1207)

Changed

  • Filing.sections() is chunked by edgar.documents instead of the legacy parser. It backs Filing.search(), and its old backend is deleted in 6.0, so it needed a replacement rather than a deprecation. Chunks are cut at headings with each table its own chunk, capped so no chunk swallows a whole Item. Measured across 41 era-stratified fixtures the two chunkers index the same words to within 0.10%, and across 375 phrase queries mean recall@5 is unchanged at 99.2%. (bead edgartools-07lk.3)

Deprecated

  • CurrentReport.doc warns; it was the last un-warned public route into edgar.files. It returns the legacy ChunkedDocument, while CompanyReport.doc on every other report class returns the modern edgar.documents document — so the same attribute name yields two unrelated types depending on the form, and a survey starting at the base class reads .doc as already migrated. Use .document for the parsed document, or .items / report[item] for item access. Measured across 133 era-stratified fixtures and 2,128 item lookups, no internal code path reaches the legacy parser through the report classes any more. (bead edgartools-07lk.3)