Bazel PyPi Package

A Bazel macro for building Python packages and interacting with PyPi. The goal is to have the package configuration one would have in a setup.py file inside a Bazel BUILD file as a pypi_package rule. Interacting with PyPi is then done via Bazel commands, rather than the regualr commands, which are outside the build system.

Rationale

Bazel offers the py_library, py_test and py_binary rules. These are great for working inside a single application codebase. However, many Python projects are libraries, and are exported as packages in the Python Package Index - PyPi. Bazel does not offer anything for integrating with it, at the moment. As such, all the interaction is done through the regular mechanisms described in Packaging and Distributing Projects. This implies the existance of the setup.py, MANIFEST.in and .pypirc files, as well as the existance of build, dist, *.egg-info and other top-level directories which contain Python build artifacts. This state of affairs is quite messy. There are two separate build systems present, each with their configuration and each producing different forms of output clutter. Ideally we'd want only one - Bazel.

The goal of this project is to correct this state of affairs, by providing a small set of tools which encapsulate all the configuration and steps necessary for managing the interaction with the Python package index. At some point in the future, it should be integrated with regular Bazel.

Installation and Usage

Bazel doesn't yet allow importing macro libraries in the workspace. Therefore, one has to copy pypi_package.bzl in a location from which other BUILD files can load it.

The following usage example is borrowed from the tabletest package, which is a small utility library for writing neater unit-tests.

In a BUILD file, one has to first import the macro, like so:

load("/tools/pypi_package", "pypi_package")

We define a regular Python library, which will be included in the package. In general, we can have more than one such library.

py_library(
    name = "tabletest",
    srcs = ["tabletest/__init__.py"],
    visibility = ["//visibility:public"],
    srcs_version = "PY2"
)

In a manner similar to the configuration for the setup function, we then write the pypi_package rule as:

pypi_package(
    name = "tabletest_pkg",
    version = "1.0.2",
    description = "Unit testing module for table-like test, for Python 2.",
    long_description = "README.md",
    classifiers = [
        "Development Status :: 4 - Beta",
        "Environment :: Console",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: MIT License",
        "Operating System :: POSIX",
        "Programming Language :: Python :: 2",
        "Programming Language :: Python :: 2.6",
        "Programming Language :: Python :: 2.7",
        "Topic :: Software Development :: Testing",
        "Topic :: Software Development :: Libraries :: Python Modules"
    ],
    keywords = "unittest test table testing",
    url = "http://github.com/horia141/tabletest",
    author = "Horia Coman",
    author_email = "horia141@gmail.com",
    license = "MIT",
    packages = [":tabletest"],
    test_suite = "nose.collector",
    tests_require = ["nose"],
)

We must first register the package with PyPi. This is achieved by running the following binary:

bazel run //:tabletest_register -- --pypi_user=[your username] --pypi_pass=[your password]

After registering (which should be done only once, but is otherwise idempotent), we can upload the current version of the code (the result of building the py_library rule from above) to the package index via:

bazel run //:tabletest_upload -- --pypi_user=[your username] --pypi_pass=[your password]

The name of the pypi_package rule needs to end in _pkg and the prefix for it will be used to generate the _register and _upload binaries.

Each time a new version needs to be updated, the version field must be updated.

The rule tries to mimick the behavior of the setup.py file as described here, but with extra integration with Bazel, such as accepting labels for local libraries and the README.md file etc.

Interaction With Other Packages

In the install_requires and tests_require fields, one can specify the required dependencies for installing the package and for testing it. These operate at the PyPi level however, so they are specified as regular strings. However, if one depends on a repository for a package which also uses Bazel and this macro, then one can depend on that rule, rather than just specifying it as a string.

For example, the WORKSPACE file for the SDHash project, looks like this:

git_repository(
    name = "tabletest",
    remote = "https://github.com/horia141/tabletest.git",
    tag = "v1.0.3",
)

The pypi_package rule references the tabletest package like this:

pypi_package(
    name = "sdhash_pkg",
    version = "0.0.3",
    description = "Library for image hashing and deduplication.",
    long_description = "README.md",
    classifiers = [...],
    keywords = "photo image gif hash perceptual dedup deduplication fft",
    url = "http://github.com/horia141/sdhash",
    author = "Horia Coman",
    author_email = "horia141@gmail.com",
    license = "MIT",
    packages = [":sdhash"],
    install_requires = ["pillow", "numpy", "scipy"],
    test_suite = "nose.collector",
    tests_require = ["nose", "@tabletest//:tabletest_pkg"],
)

The tests_require line could also have looked like this:

...
    tests_require = ["nose", "tabletest"],
...

In both cases, the package must exist in PyPi.

Requirements

A working installation of Bazel and everything in Packaging and Distributing Projects.