A helper for continuous delivery of python packages
Python's packaging and distribution scheme is really designed for manual intervention, most notably as regards versions. In your typical project, the version is hardcoded somewhere in the source tree, and the project history will be littered with messages like "bumped version to x.y." Isn't this what tagging is for? Also, why should a work in progress be given a version number in the first place? It seems backwards. It also is solvable.
Aside from that major issue, there are a number of minor annoyances that arise when handling python
projects in github. Chief among these is the problem of README files. Github clearly prefers
README.md
files, and the markdown syntax widely available and very natural to use. Pypi, on the
other hand, requires the use of README.rst
files (or at least that format) in order to display on
the project's main site. I find this more difficult to work with and, regardless, one should be
able to use one's preferred README syntax. Converting from one to the other is doable, so it should
also be simple.
Finally, when doing any kind of modification of source files on the fly (a build process) it is
important that the package tree be untouched. This is largely due to the fact that an automated
tagging process (critical to CD) falls down when the working copy is dirty. As such, automated
management of a working directory which can be safely ignored in VCS is an integrated part of
hatchery
.
- Isolation of all source manipulations to a working directory (
.hatchery.work
) - On-the-fly version management using the popular
_version.py
specification method- Logic to make sure that only pypi-accepted (see PEP-386) versions are used
- Optional on-the-fly readme conversion from
md
torst
- Notion of "tasks" which must be run in a particular order
- Dictate as little project structure as possible (see below)
As of the time of this writing, there are a few prerequisites that a project must meet in order
to integrate with hatchery
:
- The project should consist of a single root package which contains all of the python logic. That package can be as large or as small as is needed, containing as many subpackages as you like, but there must only be one.
- There are some code requirements in
setup.py
and<packagename>._version.py
. Runhatchery check
for more information. And don't worry, if these prereqs aren't met,hatchery
will tell you about it instead of doing something wonky. - In order to make use of some of the features (running tests, registering and uploading releases of your project), you will have to maintain a configuration file. More information below.
hatchery
automatically infers the name of top-level package during operation based on the layout
of the file system, and it expects there to be exactly one. In order for this to work properly,
your project needs to look like this:
# the project directory is the path you cloned to
project_directory/
.git/
.hatchery.yml
...
# the package name can be whatever you want, so long as it's not "tests"
package_name/
__init__.py
...
# the "tests" directory is where your tests are stored, and will be ignored when determining
# the top-level package -- any other directories which contain an __init__.py and are not
# children of package_name will cause the inference to fail
tests/
__init__.py
...
So, if you want your tests in a python package, make sure it's called tests
. If there are two
python packages which are not named tests
, hatchery
will throw an error. This is in service
of a unified package version. In order for setup.py
(and setuptools
) to infer the project
version from the package safely, there should only be one package from which to infer.
$ pip install hatchery
As always, it is strongly recommended that you use this inside of a virtual environment.
As previously mentioned, in order to take full advantage of what hatchery
has to offer, there is
a little configuration that is required...and I do mean a little. There are a number of parameters
available, but most of them can just be left to their default values:
Parameter | Default value | Usage |
---|---|---|
auto_push_tag |
False |
Automatically run the tag-and-push logic after a successful upload operation |
create_wheel |
True |
Create a wheel along with the source distribution during the packaging step |
git_remote_name |
'origin' |
The name of the remote to push to when pushing a git tag |
pypi_repository |
None |
String parameter describing which pypi index server to upload packages to. It actually refers to an alias which must be defined in your pypirc file |
readme_to_rst |
True |
Convert a README.md file to README.rst on the fly if the former is detected and the latter is not. This feature requires pandoc (OS-level dependency) ... so if you do not want to depend on pandoc , set to False and this feature won't be used. |
test_command |
None |
A list of arbitrary shell commands that should be run during the test task. If any of them fails, the test will be considered a failure. |
These parameters should be defined in yaml format in the
file .hatchery.yml
in the root of your project. If you want to make any of them global across
all your projects, you can also choose to define them in ~/.hatchery/hatchery.yml
; just remember
that the project-level file's values will always win!
A minimal .hatchery.yml
might look like this:
---
pypi_repository: pypi
test_command:
- tox
See .hatchery.yml
in this repository for a contextual example.
Aside: there are lots of different opinions about how best to test one's code. There are many
frameworks, and many ways to execute them. Allowing users to have complete control over this was
a key design decision. You want to use tox
? Go for it! Prefer using py.test
directly? Fine by
me. Think pylint
is important? Throw it on there! The point is, choose what testing feedback is
important to you, and hatchery
will faithfully execute it for you.
New feature alert! If you specify a full address of a pypi server as a value for pypi_repository
,
hatchery
will attempt to upload to that server, bypassing the pypirc file entirely. This will
only work for anonymous-upload servers, and cannot be used for interaction with public servers
such as pypi.python.org! Example:
pypi_repository: https://pypi.mydomain.com
Make sure you have all of the prerequisites in place
$ hatchery check
Run all tests defined in configuration
$ hatchery clean test
Register your project with the pypi repository defined in configuration
$ hatchery register
Create packages (with markdown -> rst conversion)
$ hatchery package --release-version=1.2.3
Upload your packages to the pypi repository defined in configuration
$ hatchery upload
String everything together in one go!
$ hatchery clean register test package upload --release-version=1.2.3
Find out what other great features you're missing out on
$ hatchery help
I wrote this utility because it helps me to work in the way in which I am most productive. It will not be perfect for everyone...not yet. If you think there's something missing that would help you find your happy path, please open up a feature request. Better yet, implement it and throw up a pull request. Feedback is welcome!