Python tools to help with the development & deployment of company-private Python packages.

It was developped to use Sonatype Nexus as a private Pypi mirror (until it supports this natively), but should be adaptable to any repository supporting artifact upload with HTTP.

It is composed of 2 distincts tools :

• nexus_uploader : a simple module to be used as a setup_requires entry in setup.py, in order to easily upload Python packages onto a Nexus.
• pyRequirements2nexus : a CLI tool to convert standard Python requirements.txt into nexus-requirements.txt files, made of URLs pointing to installable packages that pyRequirements2nexus mirrored on your Nexus.

Table of Contents ================

Workflow diagram

Features

• fully compatible Python 2 & 3

nexus_uploader features

• easy integration in setup.py
• HTTP BasicAuth to connect to Sonatype Nexus

pyRequirements2nexus features

• full dependency resolution of Python packages, working with both Pypi public ones & private ones on a Nexus
• to install packages, the end machine only require a connexion to the Nexus host, not to the Internet
• support -e editable packages in requirements.txt
• support package URL fallbacks as comments in requirements.txt
• can select only Python 2 packages with --allowed-pkg-classifiers py2.py3-none-any,py2-none-any
• a list of packages already included in the base environment can be specified (e.g. if you are using Anaconda), so that they will be excluded from the final nexus-requirements.txt

Limitations

• only support == version locking in requirements.txt (not >=). This is intentional, to ensure package versions do not change unexpectedly.

How to upload home-made Python packages to my Nexus ?

Here is a handy recipe you can put in your setup.py :

setup(
    ...
    setup_requires=['nexus_uploader']
)

Usage:

$ python setup.py sdist nexus_upload --repository http://nexus/content/repositories/Snapshots/com/vsct/my_project --username $REPOSITORY_USER --password $REPOSITORY_PASSWORD

Important note

Contrary to requirements.txt files, there is no way to include URLs to dependencies in setup.py install_requires (nor setup_requires). To work around this limitation, pyrequirements2nexus takes advantage of the deprecated "dependency_links" mechanism. Using them, it is able to discover the full chain of Python packages dependencies, both from Pypi AND from your local Nexus.

To easily generate the "dependency_links.txt" in your Nexus-hosted private package out of your requirements.txt, use the following recipe in your its setup.py :

with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')) as requirements_file:
    requirements = requirements_file.readlines()
    dependency_links = [req for req in requirements if req.startswith('http')]
    install_requires = [req for req in requirements if not req.startswith('http')]

setup(
    ...
    install_requires=install_requires,
    dependency_links=dependency_links,
)

Because "dependency_links" are not supported since pip 1.6, they will NOT be installed by pip install normal recursive dependency-retrieval algorithm. You should always use the flat nexus-requirements.txt with pip install.

pyRequirements2nexus usage

pip install nexus_uploader
pyRequirements2nexus --help

Also take a look at jenkins-install-python-requirements.sh for an example of how we use it on our Jenkins.

Installation of nexus-requirements.txt on an end machine

pip install --user --no-index --no-deps --no-cache-dir --upgrade --requirement nexus-requirements.txt

Supported requirements.txt format

http://nexus/content/repositories/repo_id/my/project/group/mypkgname/1.0/mypkgname-1.0-py2.py3-none-any.tar.gz
nose==1.3.7   # -> transformed into an URL like this: http://nexus/content/repositories/repo_id/my/project/group/...
-e ../my/relative/path  # http://nexus/content/repositories/...fallback_url...

Contributing

pre-commit hooks installation:

pip install -r dev-requirements
pre-commit install

Unit tests:

py.test tests/

Smoke tests using Pypi:

ipython3 --pdb tests/smoke_test_extract_classifier_and_extension.py 200

FAQ

pip install - Download error on https://pypi.python.org / Couldn't find index page for

The stack trace :

Collecting http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (from -r scripts/requirements.pip (line 12))
  Downloading http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (50kB)
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found!
    Couldn't find index page for 'vcversioner' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
    No local packages or download links found for vcversioner

Explanation : python-jsonschema/jsonschema#276

Solution :

$ cat <<EOF > ~/.pydistutils.cfg
[easy_install]
allow_hosts = nexus
find_links = http://nexus/content/repositories/pip/com/vsct/pip/vcversioner/2.14.0.0/
EOF

How to generate a "--default-packages" file out of an Anaconda .sh installer

grep -aF 'extract_dist ' Anaconda3-2.4.1-Linux-x86_64.sh \
    | perl -p -e 's/extract_dist (.+?[0-9])[^.]*$/\1\n/;' -e 's/^(.+)-(.+)$/\1 == \2/;' \
    | grep -vE '^(_cache|_license|anaconda|python) ' > anaconda3-2.4.1_included_packages.txt

Tip for easily removing packages from your nexus

pip install --user repositorytools
export REPOSITORY_USER=...
export REPOSITORY_PASSWORD=
artifact delete http://nexus/content/repositories/pip/com/vsct/pip/ultrajson/1.35/ultrajson-1.35-macosx-10.6-intel.tar.gz

ToDo

• detect if package releases on Pypi require gcc compilation (are they using setuptools/distutils Extension in setup.py ?)
• classifier-based selection of Python packages
• add support for md5 & sha1 upload/checks