AO3Fetch

Tool for scraping the work URLs off of any AO3 page. Capable of navigating the depths of index pages. Designed for use with the FanFicFare extension for Calibre.

Notes for AO3 Maintainers

• This crawler uses the user-agent string legowerewolf-ao3scraper/[commit].
• The crawler enforces a maximum request rate of 1 request per 10 seconds.
• It will also obey Retry-After headers if they are set in the response.
• I am more than happy to make changes if requested.

Arguments

• -url (string) URL to start crawling from
• -delay (int) Delay between requests in seconds (default 10)
• -login (string) Login credentials in the form of username:password
• -pages (int) Number of pages to crawl (default 1)
• -progress (boolean) Show progress bar (default true, disable with -progress=false)
• -series (boolean) Include series in the crawl (default true, disable with -series=false)
• -version (boolean) Show version and exit (default false)
• -outputFile (string) Write the list of collected works to a file instead of the terminal output.