+ 
+ 
+ 
+ 
Browser history or bookmarks exports (Chrome, Firefox, Safari, IE, Opera, and more)
+ - 
RSS, XML, JSON, CSV, SQL, HTML, Markdown, TXT, or any other text-based format
+ - 
+ 
The aim of ArchiveBox is to go beyond what the Wayback Machine and other public archiving services can do, by adding a headless browser to replay sessions accurately, and by automatically extracting all the content in multiple redundant formats that will survive being passed down to historians and archivists through many generations.
+
+ #### User Interface & Intended Purpose
+
+ ArchiveBox differentiates itself from [similar projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by being a simple, one-shot CLI interface for users to ingest bulk feeds of URLs over extended periods, as opposed to being a backend service that ingests individual, manually-submitted URLs from a web UI. However, we also have the option to add urls via a web interface through our Django frontend.
+
+ #### Private Local Archives vs Centralized Public Archives
+
+ Unlike crawler software that starts from a seed URL and works outwards, or public tools like Archive.org designed for users to manually submit links from the public internet, ArchiveBox tries to be a set-and-forget archiver suitable for archiving your entire browsing history, RSS feeds, or bookmarks, ~~including private/authenticated content that you wouldn't otherwise share with a centralized service~~ (do not do this until v0.5 is released with some security fixes). Also by having each user store their own content locally, we can save much larger portions of everyone's browsing history than a shared centralized service would be able to handle.
+
+ #### Storage Requirements
+
+ Because ArchiveBox is designed to ingest a firehose of browser history and bookmark feeds to a local disk, it can be much more disk-space intensive than a centralized service like the Internet Archive or Archive.today. However, as storage space gets cheaper and compression improves, you should be able to use it continuously over the years without having to delete anything. In my experience, ArchiveBox uses about 5gb per 1000 articles, but your milage may vary depending on which options you have enabled and what types of sites you're archiving. By default, it archives everything in as many formats as possible, meaning it takes more space than a using a single method, but more content is accurately replayable over extended periods of time. Storage requirements can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by setting `SAVE_MEDIA=False` to skip audio & video files.
+
+ ## Learn more
+
+ Whether you want to learn which organizations are the big players in the web archiving space, want to find a specific open-source tool for your web archiving need, or just want to see where archivists hang out online, our Community Wiki page serves as an index of the broader web archiving community. Check it out to learn about some of the coolest web archiving projects and communities on the web!
+
+ 
+
+ - [Community Wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+ - [The Master Lists](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#The-Master-Lists)
+ _Community-maintained indexes of archiving tools and institutions._
+ - [Web Archiving Software](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects)
+ _Open source tools and projects in the internet archiving space._
+ - [Reading List](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Reading-List)
+ _Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._
+ - [Communities](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Communities)
+ _A collection of the most active internet archiving communities and initiatives._
+ - Check out the ArchiveBox [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) and [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
+ - Learn why archiving the internet is important by reading the "[On the Importance of Web Archiving](https://parameters.ssrc.org/2018/09/on-the-importance-of-web-archiving/)" blog post.
+ - Or reach out to me for questions and comments via [@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) or [@theSquashSH](https://twitter.com/thesquashSH) on Twitter.
+
+ ---
+
+ # Documentation
+
+ 
+
+ We use the [Github wiki system](https://github.com/ArchiveBox/ArchiveBox/wiki) and [Read the Docs](https://archivebox.readthedocs.io/en/latest/) (WIP) for documentation.
+
+ You can also access the docs locally by looking in the [`ArchiveBox/docs/`](https://github.com/ArchiveBox/ArchiveBox/wiki/Home) folder.
+
+ ## Getting Started
+
+ - [Quickstart](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart)
+ - [Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install)
+ - [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker)
+
+ ## Reference
+
+ - [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage)
+ - [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration)
+ - [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
+ - [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site)
+ - [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving)
+ - [Publishing Your Archive](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive)
+ - [Chromium Install](https://github.com/ArchiveBox/ArchiveBox/wiki/Install-Chromium)
+ - [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview)
+ - [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
+ - [Python API](https://docs.archivebox.io/en/latest/modules.html)
+ - REST API (coming soon...)
+
+ ## More Info
+
+ - [Tickets](https://github.com/ArchiveBox/ArchiveBox/issues)
+ - [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap)
+ - [Changelog](https://github.com/ArchiveBox/ArchiveBox/wiki/Changelog)
+ - [Donations](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
+ - [Background & Motivation](https://github.com/ArchiveBox/ArchiveBox#background--motivation)
+ - [Web Archiving Community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
+
+ ---
+
+ # ArchiveBox Development
+
+ All contributions to ArchiveBox are welcomed! Check our [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and [Roadmap](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap) for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap.
+
+ ### Setup the dev environment
+
+ First, install the system dependencies from the "Bare Metal" section above.
+ Then you can clone the ArchiveBox repo and install
+ ```python3
+ git clone https://github.com/ArchiveBox/ArchiveBox
+ cd ArchiveBox
+ git checkout master # or the branch you want to test
+ git pull
+
+ # Install ArchiveBox + python dependencies
+ python3 -m venv .venv && source .venv/bin/activate && pip install -e .[dev]
+ # or
+ pipenv install --dev && pipenv shell
+
+ # Install node dependencies
+ npm install
+
+ # Optional: install the extractor dependencies
+ ./bin/setup.sh
+
+ # Optional: develop via docker by mounting the code dir into the container
+ # if you edit e.g. ./archivebox/core/models.py on the docker host, runserver
+ # inside the container will reload and pick up your changes
+ docker build . -t archivebox
+ docker run -it -p 8000:8000 \
+ -v $PWD/data:/data \
+ -v $PWD/archivebox:/app/archivebox \
+ archivebox server 0.0.0.0:8000 --debug --reload
+ ```
+
+ ### Common development tasks
+
+ See the `./bin/` folder and read the source of the bash scripts within.
+ You can also run all these in Docker. For more examples see the Github Actions CI/CD tests that are run: `.github/workflows/*.yaml`.
+
+ #### Run the linters
+
+ ```bash
+ ./bin/lint.sh
+ ```
+ (uses `flake8` and `mypy`)
+
+ #### Run the integration tests
+
+ ```bash
+ ./bin/test.sh
+ ```
+ (uses `pytest -s`)
+
+ #### Build the docs, pip package, and docker image
+
+ ```bash
+ ./bin/build.sh
+
+ # or individually:
+ ./bin/build_docs.sh
+ ./bin/build_pip.sh
+ ./bin/build_docker.sh
+ ```
+
+ #### Roll a release
+
+ ```bash
+ ./bin/release.sh
+ ```
+ (bumps the version, builds, and pushes a release to PyPI, Docker Hub, and Github Packages)
+
+
+ ---
+
+ 
+ 
+ ArchiveBox
+ {{ title }}
{% endif %} #}{% endblock %} + {% block content %} + {% block object-tools %}{% endblock %} + {{ content }} + {% endblock %} + {% block sidebar %}{% endblock %} ++
+ ArchiveBox
The open-source self-hosted web archive.
+
+ ▶️ Quickstart |
+ Demo |
+ Github |
+ Documentation |
+ Info & Motivation |
+ Community |
+ Roadmap
+
+ + "Your own personal internet archive" (网站存档 / 爬虫) ++ + + +
+ 
+ 
+ 
+ 
+ 
+
+ +
