sec_filings: fetch from EDGAR directly to fix stale data

[peterhxk]

Proposal: EDGAR backend for sec_filings to fix incomplete filing coverage

Problem

Ticker.get_sec_filings() sources filings from Yahoo's quoteSummary API (secFilings module). The dates Yahoo reports are accurate for the filings it carries, but the list itself is badly incomplete in two ways.

1. Ownership and insider filings are missing entirely. Forms 3, 4, and 144 and Schedules 13D/G never appear, for any ticker I tested. Insider transaction data is one of the most common reasons to want SEC filings programmatically, and it is simply not available through yfinance today.

2. Core filings are silently dropped when several land on the same day. Yahoo's history is deep (multi-year, back to 2021 for some tickers I checked), but it appears to keep at most one filing per date. Every same-day collision I found follows this pattern, four days across two tickers:

• ETSY 2026-04-29: 10-Q and 8-K filed; Yahoo carries only the 10-Q (missing 8-K: 0001370637-26-000042)
• ETSY 2026-02-19: 10-K and 8-K filed; Yahoo carries only the 10-K (missing 8-K: 0001370637-26-000017)
• MSFT 2026-04-29: 10-Q and 8-K filed; Yahoo carries only the 10-Q
• MSFT 2026-01-28: 10-Q, earnings 8-K, and IRANNOTICE all filed; Yahoo carries only the IRANNOTICE, dropping both the quarterly report (0001193125-26-027207) and the earnings release
  Across all raw Yahoo output I inspected, no date ever appears twice. Because earnings press-release 8-Ks routinely coincide with the 10-Q or 10-K, this behavior systematically drops earnings releases, and as the MSFT case shows, it can drop the 10-Q itself. Which filing survives the collision does not appear to follow any sensible priority.

Measured over the last 180 days (run on 2026-07-21 with the script attached below):

Ticker	Filings via get_sec_filings()	Filings on EDGAR	Notable gaps
AAPL	7	43	26 Form 4s, 6 Form 144s, 13G, PX14A6G
MSFT	7	83	63 Form 4s, and one of two 10-Qs missing*
TSLA	9	28	9 Form 4s, 6 Form 144s, 4 Schedule 13G/As
ETSY	7	106	61 Form 4s, and two of six 8-Ks missing
CAVA	8	79	39 Form 4s, 22 Form 144s, Forms 3
RDDT	6	82	58 Form 4s; newest filing 5 weeks ahead of Yahoo's

* MSFT's missing 10-Q (0001193125-26-027207, filed 2026-01-28) collided with the earnings 8-K and an IRANNOTICE; Yahoo kept only the IRANNOTICE. Details in problem 2 above.

There is also a related note already in the codebase: _fetch_calendar() in scrapers/quote.py says "secFilings return too old data, so not requesting it for now." In my tests the dates on covered filings were accurate, so the current problem is coverage rather than staleness, but it suggests this module's quality has been a known concern.

Proposal

Keep the existing interface and add an opt-in backend that queries SEC EDGAR directly:

ticker.get_sec_filings()                  # unchanged: Yahoo (backward compatible)
ticker.get_sec_filings(source="edgar")    # new: complete filing history from the SEC

EDGAR's data.sec.gov/submissions/ endpoint is free and keyless, includes every filing (insider forms included), and posts filings within minutes of SEC acceptance. It also returns data in columnar form, so this pairs naturally with the DataFrame conversion that already exists as commented out "Experimental" code in _fetch_sec_filings(): for example a pd.DataFrame return for the EDGAR path, or an as_dict= flag matching the convention used by get_major_holders() and others.

How it works

Two requests, no auth:

1. Ticker to CIK mapping from https://www.sec.gov/files/company_tickers.json, fetched once and cached, reusing the existing cache.py layer the same way the timezone cache does.
2. Filings index from https://data.sec.gov/submissions/CIK{cik:010d}.json.
   Prototype:

import pandas as pd
 
def _fetch_sec_filings_edgar(self) -> pd.DataFrame:
    cik = self._get_cik()  # from cached ticker->CIK map
    url = f"https://data.sec.gov/submissions/CIK{cik:010d}.json"
    j = self._data.cache_get(url=url, headers=EDGAR_HEADERS).json()
    df = pd.DataFrame(j["filings"]["recent"])
    df["filingDate"] = pd.to_datetime(df["filingDate"])
    acc = df["accessionNumber"].str.replace("-", "", regex=False)
    df["url"] = ("https://www.sec.gov/Archives/edgar/data/"
                 + str(cik) + "/" + acc + "/" + df["primaryDocument"])
    return df.set_index("filingDate")

This yields every recent filing (10-K, 10-Q, 8-K, S-1, Form 4, and so on) with form type, report date, accession number, and a direct document URL. A form= filter would let users grab, say, only insider filings in one line.

Anticipated concerns (and how I'd address them)

"yfinance is a Yahoo library, a second data source is scope creep." Fair, and I'd understand a no on these grounds. The counterargument: this adds no new API surface, it fixes a measured deficiency behind an existing method, and it is opt-in only. Yahoo remains the default. If the maintainers prefer, I'm equally happy to ship this as a small companion package instead. I'd just rather ask first, since the coverage gap above affects the flagship forms and not only exotic ones.

SEC fair-access rules. EDGAR requires a descriptive User-Agent with contact info and caps clients at 10 requests per second; violators get IP banned. The implementation would (a) let users set their contact string via config, refusing EDGAR requests without one, and (b) rate limit well below the cap. This proposal makes zero additional requests to Yahoo, so it cannot affect yfinance's standing with its primary upstream.

Older filings. The submissions endpoint inlines roughly the most recent 1,000 filings; older ones live in paginated supplement files referenced in the same JSON. v1 would cover the recent set (which is already vastly more than Yahoo returns) and document the limit, with pagination as a follow-up if wanted.

Ticker edge cases. The CIK map covers SEC registrants only. Foreign issuers without US filings and most non-equity tickers would return an empty result with a clear message rather than an error.

What I'm offering

If there's appetite for this, I'll open a PR with: the CIK mapping and caching, the EDGAR fetcher with rate limiting and mandatory User-Agent config, form type filtering, tests, and docs. If the consensus is "out of scope," I'll build it as a companion package and would appreciate at most a pointer in the docs.

Would love feedback on: (1) in scope or not, (2) source="edgar" kwarg versus a separate method like get_edgar_filings(), (3) dict versus DataFrame for the new path's return type.

Reproduction script

```python import datetime as dt from collections import Counter

import requests
import yfinance as yf

CONTACT = "Your Name your@email.com" # SEC fair-access requirement
WINDOW_DAYS = 180
TICKERS = ["AAPL", "MSFT", "TSLA", "ETSY", "CAVA", "RDDT"]

EDGAR_HEADERS = {"User-Agent": CONTACT}
CUTOFF = dt.date.today() - dt.timedelta(days=WINDOW_DAYS)

def cik_map():
r = requests.get("https://www.sec.gov/files/company_tickers.json",
headers=EDGAR_HEADERS, timeout=30)
r.raise_for_status()
return {v["ticker"]: int(v["cik_str"]) for v in r.json().values()}

def edgar_recent(cik):
url = f"https://data.sec.gov/submissions/CIK{cik:010d}.json"
r = requests.get(url, headers=EDGAR_HEADERS, timeout=30)
r.raise_for_status()
recent = r.json()["filings"]["recent"]
return sorted(((dt.date.fromisoformat(d), f)
for f, d in zip(recent["form"], recent["filingDate"])
if dt.date.fromisoformat(d) >= CUTOFF), reverse=True)

def yahoo_recent(ticker):
return sorted(((f["date"], f.get("type", "?"))
for f in yf.Ticker(ticker).get_sec_filings()
if f["date"] >= CUTOFF), reverse=True)

def summarize(name, rows):
if not rows:
return f"{name}: nothing in window"
forms = Counter(f for _, f in rows)
return (f"{name}: latest = {rows[0][1]} on {rows[0][0]} | "
f"{len(rows)} filings in {WINDOW_DAYS}d | "
+ ", ".join(f"{k}x{v}" for k, v in sorted(forms.items())))

ciks = cik_map()
for t in TICKERS:
print(f"\n=== {t} ===")
print(summarize("Yahoo", yahoo_recent(t)))
print(summarize("EDGAR", edgar_recent(ciks[t])))

 
</details>

[peterhxk]

@ranaroussi I'm not sure who I should reach out to regarding this, I noticed that the discussion board is quite inactive, should I directly implement this and open a PR for review?

[ValueRaider]

yfinance strictly only fetches from Yahoo