Complete rewrite of package to use EDGAR Search API (#52)

* Add type annotations to existing code before rewrite.

* Solid progress toward rewrite.

* More progress toward rewrite.

* More progress toward rewrite.

* Full p39 support

* Add tests to get coverage to 100% and improve error handling.

* Update tests

* Update tests

* Added search query functionality

* Added search query functionality

* Fixed linting and added supported filings to README.

* Rate-limit API requests and add CHANGELOG and migration guide.

* Updated docs and README

* Added to CHANGELOG/migration guide; added test for URL resolution.

* Remove lstrip of 0s since search API does not allow for truncated CIKs

.github/workflows/continuous_integration.yml

@@ -11,13 +11,13 @@ on:
 jobs:
   test:
     name: ${{ matrix.os }} (${{ matrix.python-version }})
-    runs-on: ${{ matrix.os }}-latest
+    runs-on: ${{ matrix.os }}
 
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9-dev"]
-        os: [Ubuntu, MacOS, Windows]
+        python-version: ["3.6", "3.7", "3.8", "3.9"]
+        os: [ubuntu-latest, macos-latest, windows-latest]
     steps:
       - uses: actions/checkout@v2
       - name: Set up Python ${{ matrix.python-version }}
@@ -40,10 +40,10 @@ jobs:
 
     steps:
       - uses: actions/checkout@v2
-      - name: Set up Python 3.8
+      - name: Set up Python 3.9
         uses: actions/setup-python@v2
         with:
-          python-version: "3.8"
+          python-version: "3.9"
       - name: Install dependencies
         run: |
           pip install -U pip setuptools wheel

.gitignore

@@ -1,3 +1,5 @@
+sec-edgar-filings/
+
 # small local test file for sanity checks
 sanity_check.py
 

.pre-commit-config.yaml

@@ -2,12 +2,12 @@ default_language_version:
   python: python3
 repos:
   - repo: https://github.com/PyCQA/bandit
-    rev: 1.6.2
+    rev: 1.7.0
     hooks:
       - id: bandit
         args: [-lll, --quiet, --exclude=tests/**]
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v3.2.0
+    rev: v3.4.0
     hooks:
       - id: check-case-conflict
       - id: check-docstring-first
@@ -19,28 +19,28 @@ repos:
       - id: fix-encoding-pragma
         args: [--remove]
       - id: trailing-whitespace
-        exclude: ^tests/sample_filings/
+        exclude: ^tests/sample-filings/
   - repo: https://github.com/timothycrosley/isort
-    rev: 5.4.2
+    rev: 5.7.0
     hooks:
       - id: isort
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.7.2
+    rev: v2.7.4
     hooks:
       - id: pyupgrade
         args: [--py36-plus]
   - repo: https://github.com/pre-commit/pygrep-hooks
-    rev: v1.6.0
+    rev: v1.7.0
     hooks:
       - id: python-no-eval
       - id: rst-backticks
   - repo: https://github.com/psf/black
     rev: 20.8b1
     hooks:
       - id: black
-        args: [--safe, --quiet]
+        args: [--quiet, --target-version=py36]
   - repo: https://gitlab.com/pycqa/flake8
-    rev: 3.8.1
+    rev: 3.8.4
     hooks:
       - id: flake8
         additional_dependencies: [flake8-bugbear]

CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## 4.0.0 - 1/16/2021
+
+This is a major breaking release. Please see the [v4 migration guide](https://github.com/jadchaar/sec-edgar-downloader/blob/master/docs/v4_migration_guide.md) for information on how to upgrade and adapt your existing codebase to the new package version.
+
+### Added
+
+- The [SEC Edgar Full-Text Search API](https://www.sec.gov/edgar/search/) is now used to fetch filing download URLs. This approach replaces the existing fragile scraping and ATOM RSS implementation used in existing versions of this package.
+  - Note: this API only allows for fetching filings after December 1, 2000.
+- Added support for searching and downloading filings via a `query` kwarg:
+```python
+dl = Downloader()
+# Download all Apple proxy statements that contain the word "antitrust"
+dl.get("DEF 14A", "AAPL", query="antitrust")
+```
+- Filing details, whose extensions vary based on filing type, are now downloaded in addition to the full submission `txt` file. See the migration guide for information on the revamped folder structure.
+- Added the ability to download all available SEC filings. Please see the [README](https://github.com/jadchaar/sec-edgar-downloader/blob/master/README.rst) for a full list of all supported filings.
+- `Path` objects can now be used to specify a `download_folder` when constructing a `Downloader` object.
+- Added type annotations throughout the codebase.
+
+### Changed
+
+- The current working directory is now used as the default download location. Custom paths can be specified when constructing `Downloader` objects.
+- All arguments passed to `dl.get()` other than `filing` and `ticker_or_cik` must be used with a keyword:
+```python
+dl = Downloader()
+dl.get(
+    "10-K",
+    "AAPL",
+    # All other arguments must be used with a keyword
+    amount=1,
+    after="2019-01-01",
+    before="2021-01-01",
+    include_amends=True,
+    download_details=True,
+    query="sample query"
+)
+```
+- The `after_date`, `before_date`, and `num_filings_to_download` kwargs have been renamed to `after`, `before`, and `amount`, respectively.
+
 ## 3.0.5 - 5/16/2020
 
 ### Added

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) [2019] [Jad Chaar]
+Copyright (c) [2020] [Jad Chaar]
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Makefile

@@ -5,8 +5,9 @@ auto: build38
 build36: PYTHON_VER = python3.6
 build37: PYTHON_VER = python3.7
 build38: PYTHON_VER = python3.8
+build39: PYTHON_VER = python3.9
 
-build36 build37 build38: clean
+build36 build37 build38 build39: clean
 	$(PYTHON_VER) -m venv venv
 	. venv/bin/activate; \
 	pip install -U pip setuptools wheel; \