You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Complementary to #88, but separate because the download script runs differently. Iterating over the pages of existing sources takes a non-negligible amount of time before the download loop begins. Once it does, it is possible to download a few thousand sources per hour.
Feature Summary
To prepare for the need to download large source lists, we should streamline scope_download_classification.py to run as quickly as possible.
Implementation details
The page-by-page loop before downloads begin, along with the download loop that starts afterwards, are both good areas to focus for this enhancement.
The text was updated successfully, but these errors were encountered:
With #90 implemented, if the user specifies a group from which to download rather than individual sources, it now takes a few minutes to download a few thousand sources from that group.
Complementary to #88, but separate because the download script runs differently. Iterating over the pages of existing sources takes a non-negligible amount of time before the download loop begins. Once it does, it is possible to download a few thousand sources per hour.
Feature Summary
To prepare for the need to download large source lists, we should streamline scope_download_classification.py to run as quickly as possible.
Implementation details
The page-by-page loop before downloads begin, along with the download loop that starts afterwards, are both good areas to focus for this enhancement.
The text was updated successfully, but these errors were encountered: