Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add download option to skip accession numbers #142

Merged
merged 6 commits into from
Jul 26, 2024
Merged

Add download option to skip accession numbers #142

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
96ffc70
Add download option to skip accession numbers
spolcyn May 2, 2024
dee72ad
Use `Set` from typing and improve var name
spolcyn Jun 20, 2024
c5d826c
Fix typing annotation
spolcyn Jun 20, 2024
378443c
Add test for skipping accession numbers
spolcyn Jun 20, 2024
e11e67f
Apply linter
spolcyn Jun 20, 2024
b9aa251
Merge branch 'master' into master
spolcyn Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions sec_edgar_downloader/_Downloader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ def get(
before: Optional[Date] = None,
include_amends: bool = False,
download_details: bool = False,
skip_accession_numbers: Optional[set[str]] = None,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use Set from typing library here :)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO we should not expose this as a parameter, but instead automatically use the existing file system to determine which items to skip.

Thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That solution works well for when everything is done on a single server, and indeed it worked well for me until I started decoupling storage and computation. Once that happened, I needed an efficient way to let the ephemeral downloading jobs -- which have only small, temporary file systems -- know what had already been downloaded, which led me here.

) -> int:
"""Download filings and save them to disk.

Expand All @@ -84,6 +85,7 @@ def get(
Defaults to False.
:param download_details: denotes whether to download human-readable and easily
parseable filing detail documents (e.g. form 4 XML, 8-K HTML). Defaults to False.
:param skip_accession_numbers: Set of accession numbers to skip when downloading.
:return: number of filings downloaded.

Usage::
Expand Down Expand Up @@ -173,6 +175,7 @@ def get(
download_details,
# Save ticker if passed in to form file system path for saving filings
ticker=ticker_or_cik if not is_cik(ticker_or_cik) else None,
skip_accession_numbers=skip_accession_numbers,
spolcyn marked this conversation as resolved.
Show resolved Hide resolved
),
self.user_agent,
)
Expand Down
3 changes: 3 additions & 0 deletions sec_edgar_downloader/_orchestrator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,9 @@ def get_to_download(cik: str, acc_num: str, doc: str) -> ToDownload:
def fetch_and_save_filings(download_metadata: DownloadMetadata, user_agent: str) -> int:
successfully_downloaded = 0
to_download = aggregate_filings_to_download(download_metadata, user_agent)
if download_metadata.skip_accession_numbers is not None:
to_download = [td for td in to_download if td.accession_number not in download_metadata.skip_accession_numbers]

for td in to_download:
try:
save_location = get_save_location(
Expand Down
1 change: 1 addition & 0 deletions sec_edgar_downloader/_types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ class DownloadMetadata:
include_amends: bool = False
download_details: bool = False
ticker: Optional[str] = None
skip_accession_numbers: Optional[set[str]] = None


@dataclass
Expand Down
Loading