PyPIM

PyPI Intelligent Mirror

Introduction

There are many mirroring tools for PyPI.org. The well-known Bandersnatch works fine but cannot ensure dependencies.

That's why PyPIM has been written :

• whitelist
• blacklist
• filter by platforms
• retain only recent releases
• respect the dependencies

But computing dependencies is time and space comsuming. The whole package metadata are required and easy to reach.

Using PyPIM

Usage: pypim.py [OPTIONS]

Options:
  -v, --verbose       verbose mode
  -nv, --non-verbose  no so much verbose
  -n, --dry-run       dry run
  -u, --update        update list of packages
  -m, --metadata      download JSON metadata
  -p, --packages      mirror packages
  -a, --add TEXT      package name
  -h, --help          Show this message and exit.

Fetch the list of packages

./pypim.py -u

The script uses the XML-RPC API to fetch the list of packages and their last_serial (a growing-only internal counter).

Download the package metadata

Based on the previous list of packages, the script uses the [JSON API](

./pypim.py -m

The script uses the XML-RPC API to download the package metadata.

The response, in JSON, is made of 4 sections:

• info : package description
• last_serial : a number
• releases : list of releases
• urls: files of the current releases

The metadata is stored in both raw and decoded formats, in two SQLite3 databases.

Download the packages

./pypim.py -p [-a name1] [-a name2[==version]]...

Exclude packages

Filter by platforms

Keep only recents releases

Respect the dependencies

Add a whitelist

Examples

Sync the test index

./pypim.py -v --db test.db --test -um

Sync the whole index with our package requirements

./pypim.py -lf mirror.log -ump -a pyxb==1.2.3 -r <(pip3 freeze) --web /data/mirrors/pypi

Some reference links

• PEP 426 -- Metadata for Python Software Packages 2.0
• PEP 566 -- Metadata for Python Software Packages 2.1
• Warehouse : web application that runs PyPI
• PIP configuration file
• PyPI API