pip and friends: Packaging
(largely borrowed from Zamboni)
Your app will depend on lots of tasty open source Python libraries. The list of all your dependencies should exist in several places:
- As a submodule of vendor or vendor-local
Ultimately your app code will run against the libraries under vendor/vendor-local via mod_wsgi.
Why requirements? For development, you can use virtualenvs and pip to have a tidy self-contained environment. If run from the Django runserver command, you don't even need a web server.
The vendor library
/vendor library is supposed to contain all packages and repositories.
It enables the project to be deployed as one package onto many machines,
without relying on PyPI-based installations on each target machine.
By default, the vendor lib is checked out as a git submodule under
vendor/. If you do need to check it out separately, do:
git clone --recursive git://github.com/mozilla/playdoh-lib.git ./vendor
Once the playdoh-lib repo has been downloaded to
/vendor, you only need to
install the compiled packages (as defined in
These can come from your system package manager or from:
pip install -r requirements/compiled.txt
The vendor-local library
/vendor-local directory is laid out exactly like vendor and works the
same way. All of your custom pure Python dependencies should go here so that
you can freely merge with playdoh's vendor directory if necessary. See the
section on adding new packages below.
Global vs. local library
Playdoh provides its default library in the
vendor/ directory. You may
fork and change it, but that will make it hard to pull updates from the
upstream library later.
If you want to make only a few local additions or override some of the
vendor/, make those changes to the directory
instead, which (in
manage.py) is given precedence over playdoh's vendor
compiled.txt vs prod.txt
If a Python library requires compilation, it should be recorded in compiled.txt. These aren't as portable and cannot be shipped in the vendor library. For local development, it's nice to pip install these into a virtualenv. A common practise is to use virtualenv only for compiled libraries and vendor for the rest of your dependencies.
Adding new packages
If we wanted to add a new dependency called
cheeseballs to playdoh, you
would add it to
requirements/prod.txt. If your library isn't used in
production, then put it in
requirements/dev.txt. This makes it available
to users installing into virtualenvs.
We also need to add the new package to the vendor-local lib, since that is what runs in production...
First, we need to add the source. There are two ways, depending on how this project is hosted:
Non-git based repos (hg, CVS, tarball)
For such repos or for packages coming from PyPI, do:
pip install -I --install-option="--home=`pwd`/vendor-local" cheeseballs cd vendor-local git add lib/python/cheeseballs git commit
Optionally, if the package installs executables add them too. For example:
cd vendor-local git add bin/cheeseballer git commit
For hg repos that are not on PyPI, they can be installed with pip too
but omit the
--home option and use the
--src instead. For
pip install -I --src='vendor-local/src' \ -e hg+http://bitbucket.org/jespern/django-piston@default#egg=django-piston cd vendor-local git add src/django-piston git commit
Installed source packages need to be appended to
vendor-local/vendor.pth. See note below. For example:
echo src/django-piston >> vendor-local/vendor.pth
For a git-based package, add it as a git submodule:
cd vendor-local git submodule add git://github.com/mozilla/cheeseballs.git src/cheeseballs git commit vendor.pth .gitmodules src/cheeseballs
Further, you then need to update
vendor-local/vendor.pth. Python uses
.pth files to dynamically add directories to
The file format is simple. Consult
vendor/vendor.pth for reference.
Some packages (like
selenium) are troublesome, because
their source lives inside an extra subdirectory
src/ inside their checkout.
So they need to be sourced with
src/html5lib/src, for example. Hopefully
you won't hit any snags like that.
./manage.py shell and then
import cheeseballs to make sure
Testing Your Vendor Change
It's critical that you test your app running under mod_wsgi. Although you may use runserver day to day, go ahead and run some code through WSGI to prove vendor is setup properly. (throw an import into your view, etc)
TODO [automate these instructions](<https://github.com/mozilla/playdoh/issues/30)
Initial creation of the vendor library
The vendor repo was seeded with
pip install -I --install-option="--home=`pwd`/vendor" --src='vendor/src' -r requirements/dev.txt # ..delete some junk from vendor/lib/python... # Create the .pth file so Python can find our src libs. find src -type d -depth 1 >> vendor.pth # Add all the submodules. for f in src/*; do pushd $f >/dev/null && REPO=$(git config remote.origin.url) && popd > /dev/null && git submodule add $REPO $f done git add .
Adding lots of git submodules
As noted in Adding new packages, git-based packages are git submodules inside the vendor library. To set up the first batch of submodules, something like the following happened:
for f in src/* pushd $f && REPO=$(git config remote.origin.url) && popd && git submodule add $REPO $f
For reference: pip
The classical method of installing is using pip. We have our packages separated into three files:
- All packages that require (or go faster with) compilation. These can't be distributed cross-platform, so they need to be installed through your system's package manager or pip.
- The minimal set of packages you need to run zamboni in production. You
also need to get
- All the packages needed for running tests and development servers. This
With pip, you can get a development environment with:
pip install -r requirements/dev.txt -r requirements/compiled.txt