[feature]: Back up lists to the Internet Archive

[T145]

Contact Details

No response

What's your idea?

Instead of archiving lists to a local submodule, use the Internet Archive's API and libraries to backup lists and fall back to that URL if the original can't be accessed.

Code of Conduct

• I agree to follow this project's Code of Conduct

[T145]

Prepend list URLs with https://web.archive.org/save/. Submitted 200 response means it gets saved successfully.

429: Save Page limits 15 requests per minute and will rate limit otherwise. This project should be fine.
509: Active session limit reached. Again this project should be fine.

[T145]

Could detach the process so building can run alongside archival.

https://unix.stackexchange.com/questions/269805/how-can-i-detach-a-process-from-a-bash-script

The best plan is likely to perform archival before downloading the lists, catch any download errors, and retry those downloads with IA links.