Skip to content

v3.1.0: Add support for SPN2 flags

Choose a tag to compare

View all tags
@agude agude released this 06 Sep 18:04
· 74 commits to main since this release
v3.1.0
5fc5156

Release Notes:

This release focuses on giving users more control over the archiving process while making the tool smarter and more resilient. Building on the reliable SPN2 API foundation from version 3.0.0, this update introduces a suite of command-line flags for advanced API features, implements more robust error handling for temporary server issues, and makes the job polling mechanism significantly more efficient.

There are no breaking changes in this release.

What's New

  • Advanced API Control via Command-Line Flags: You can now control the behavior of the Internet Archive's capture API directly from the command line. This gives you fine-grained control over your archives. New flags include:

    • --capture-screenshot: Saves a full-page screenshot of the archived page.
    • --capture-outlinks: Automatically finds and archives all links on a page.
    • --if-not-archived-within: Skips archiving a URL if a recent capture already exists.
    • ...and many more. See archiver --help for the full list.

  • Increased Resilience to Server Errors: The archiver is now much more robust against temporary server-side issues. If the Internet Archive API returns a transient error (like a 503 Service Unavailable), the script will now automatically wait and retry the request instead of immediately failing the job. This significantly improves the success rate for large, long-running archives.

  • More Efficient and Responsive Polling: The mechanism for checking the status of pending jobs is now dynamic. It starts with short waits for quick feedback on fast jobs and gradually increases the delay (exponential backoff) for jobs that take longer. This is more respectful to the API, reduces unnecessary network traffic, and improves the overall efficiency of the script.

  • Improved Command-Line Help and Usability: The help text for all command-line arguments has been rewritten to be more descriptive and consistent. Each option is now explained with a full sentence, providing better context on what it does and why you might use it.

Assets 2
Loading