fedcode-next: Code pipeline and models to continuously collect various URLs and references to articles, other advisories

[pombredanne]

The references are often vanishing from the web. We should collect them and ideally have a mechanism to archive a snapshot. This could be using archive.org and dow not mean we must store the content ourselves

[ziadhany]

I think we should use web.archive.org to store archive URLs via their "Save Page Now" endpoint https://web.archive.org/save/<url>

https://github.com/akamhy/waybackpy/blob/master/waybackpy/save_api.py.
Since our URLs are stored in AdvisoryReference, we could add a new field archive_url to that model.

One major constraint is the rate limit of 15 requests per minute. With 250,000 URLs to process, the initial run would take about 11.5 days:

250,000 / (15 * 60 * 24) = 11.57 days