🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

Collect and revisit web pages.

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

WarcDB: Web crawl data as SQLite databases.

Streaming WARC/ARC library for fast web archive IO

Bitextor generates translation memories from multilingual websites

WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.

CoCrawler is a versatile web crawler built using modern tools and concurrency.

A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC

Summarize web archive capture index (CDX) files.

Command line tool to convert a file in the WARC format to a file in the ZIM format

Web archiving using Google Chrome

metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

CDXJ Indexing of WARC/ARCs

Support for writing WARC files with Scrapy

Decentralized web archiving

A whilrlwind tour of Common Crawl's data using Python

A tool for detecting viruses and NSFW material in WARC files