Skip to content
A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Fix example of platform filtering in docs (#266) Jun 4, 2019
src Move version to 3.4.1 + Update CHANGES.md Jun 18, 2019
.coveragerc Add `root_uri` configuration option. Mar 22, 2018
.gitignore Updated gitignore - ignoring test.conf Jun 21, 2018
.pre-commit-config.yaml Update Documentation about being explicit with filter plugins (#166) Mar 5, 2019
.pyup.yml [.pyup.yml] Go to invalid yaml bools + add back assignees Jun 18, 2019
.travis.yml Update Documentation about being explicit with filter plugins (#166) Mar 5, 2019
CHANGES.md Move version to 3.4.1 + Update CHANGES.md Jun 18, 2019
DEVELOPMENT.md Document + enforce we need >= 3.6.1 (#67) Oct 29, 2018
Dockerfile Add a reference Docker File (#113) Jan 5, 2019
LICENSE Add `root_uri` configuration option. Mar 22, 2018
MAINTAINERS.md Add a Maintainers guide + Mission Statement to README.md (#214) May 6, 2019
MANIFEST.in Add in support for [plugins] config section (#225) May 9, 2019
README.md Remove unmaintained links (#261) May 27, 2019
azure-pipelines.yml Update Documentation about being explicit with filter plugins (#166) Mar 5, 2019
bootstrap.sh Add `root_uri` configuration option. Mar 22, 2018
buildout.cfg Add `root_uri` configuration option. Mar 22, 2018
mypy.ini Introduce mypy and enforce it on a few files Jun 16, 2018
pytest.ini Fix Unit test Coverage + add lint run with Tox (#63) Oct 18, 2018
requirements.txt Update requirements.txt May 17, 2019
requirements_docs.txt Update requests to 2.22.0 (#238) May 17, 2019
requirements_test.txt Update tox from 3.11.0 to 3.11.1 (#237) May 16, 2019
setup.cfg [setup.cfg] Move its version 3.4.1 Jun 18, 2019
setup.py Version plugin api (#235) May 17, 2019
test_runner.py Fix turning on all plugins by default (#142) Jan 25, 2019
tox.ini Reverse to a requests happy version of urllib3 + default to testing w… Apr 25, 2019

README.md

Code style: black Build Status codecov.io Documentation Status Updates Downloads


This is a PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/.

bandersnatch maintainers are looking for more help! Please refer to our MAINTAINER documentation to see the roles and responsibilities. We would also ask you read our Mission Statement to ensure it aligns with your thoughts for this project.

  • If interested contact @cooperlees

Installation

The following instructions will place the bandersnatch executable in a virtualenv under bandersnatch/bin/bandersnatch.

  • bandersnatch requires >= Python 3.6.1

pip

This installs the latest stable, released version.

  $ python3.6 -m venv bandersnatch
  $ bandersnatch/bin/pip install bandersnatch
  $ bandersnatch/bin/bandersnatch --help

Quickstart

  • Run bandersnatch mirror - it will create an empty configuration file for you in /etc/bandersnatch.conf.
  • Review /etc/bandersnatch.conf and adapt to your needs.
  • Run bandersnatch mirror again. It will populate your mirror with the current status of all PyPI packages. Current mirror package size can be seen here: https://pypi.org/stats/
  • A blacklist or whitelist can be created to cut down your mirror size. You might want to Analyze PyPI downloads to determine which packages to add to your list.
  • Run bandersnatch mirror regularly to update your mirror with any intermediate changes.

Webserver

Configure your webserver to serve the web/ sub-directory of the mirror. For nginx it should look something like this:

    server {
        listen 127.0.0.1:80;
        server_name <mymirrorname>;
        root <path-to-mirror>/web;
        autoindex on;
        charset utf-8;
    }
  • Note that it is a good idea to have your webserver publish the HTML index files correctly with UTF-8 as the charset. The index pages will work without it but if humans look at the pages the characters will end up looking funny.

  • Make sure that the webserver uses UTF-8 to look up unicode path names. nginx gets this right by default - not sure about others.

Cron jobs

You need to set up one cron job to run the mirror itself.

Here's a sample that you could place in /etc/cron.d/bandersnatch:

    LC_ALL=en_US.utf8
    */2 * * * * root bandersnatch mirror |& logger -t bandersnatch[mirror]

This assumes that you have a logger utility installed that will convert the output of the commands to syslog entries.

Maintenance

bandersnatch does not keep much local state in addition to the mirrored data. In general you can just keep rerunning bandersnatch mirror to make it fix errors.

If you want to force bandersnatch to check everything against the master PyPI:

  • run bandersnatch mirror --force-check to move status files if they exist in your mirror directory in order get a full sync.

Be aware that full syncs likely take hours depending on PyPI's performance and your network latency and bandwidth.

Operational notes

Case-sensitive filesystem needed

You need to run bandersnatch on a case-sensitive filesystem.

OS X natively does this OK even though the filesystem is not strictly case-sensitive and bandersnatch will work fine when running on OS X. However, tarring a bandersnatch data directory and moving it to, e.g. Linux with a case-sensitive filesystem will lead to inconsistencies. You can fix those by deleting the status files and have bandersnatch run a full check on your data.

Many sub-directories needed

The PyPI has a quite extensive list of packages that we need to maintain in a flat directory. Filesystems with small limits on the number of sub-directories per directory can run into a problem like this:

2013-07-09 16:11:33,331 ERROR: Error syncing package: zweb@802449
OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'

Specifically we recommend to avoid using ext3. Ext4 and newer does not have the limitation of 32k sub-directories.

Client Compatibility

A bandersnatch static mirror is compatible only to the "static", cacheable parts of PyPI that are needed to support package installation. It does not support more dynamic APIs of PyPI that maybe be used by various clients for other purposes.

An example of an unsupported API is PyPI's XML-RPC interface, which is used when running pip search.

Bandersnatch Mission

The bandersnatch project strives to:

  • Mirror all static objects of the Python Package Index (https://pypi.org/)
  • bandersnatch's main goal is to support the main global index to local syncing only
  • This will allow organizations to have lower latency access to PyPI and save bandwidth on their WAN connections and more importantly the PyPI CDN
  • Custom features and requests may be accepted if they can be of a plugin form
    • e.g. refer to the blacklist and whitelist plugins

Contact

If you have questions or comments, please submit a bug report to https://github.com/pypa/bandersnatch/issues/new

  • IRC: #bandersnatch on Freenode (You can use webchat if you don't have an IRC client)

Code of Conduct

Everyone interacting in the bandersnatch project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.

Kudos

This client is based on the original pep381client by Martin v. Loewis.

Richard Jones was very patient answering questions at PyCon 2013 and made the protocol more reliable by implementing some PyPI enhancements.

Christian Theune for creating and maintaining bandersnatch for many years!

You can’t perform that action at this time.