We're pleased that you are interested in working on Warehouse.
Your first pull request
After you set up your development environment and ensure you can run the tests and build the documentation (using the instructions in this document), take a look at :doc:`our guide to the Warehouse codebase <../application>`. Then, look at our open issues that are labelled "good first issue", find one you want to work on, comment on it to say you're working on it, then submit a pull request. Use our :doc:`submitting-patches` documentation to help.
Setting up a development environment to work on Warehouse should be a straightforward process. If you have any difficulty, contact us so we can improve the process:
- For bug reports or general problems, file an issue on GitHub;
- For real-time chat with other PyPA developers, join
- For longer-form questions or discussion, message the pypa-dev mailing list.
Detailed installation instructions
Getting the Warehouse source code
Clone the Warehouse repository from GitHub:
git clone email@example.com:pypa/warehouse.git
Configure the development environment
Docker simplifies development environment set up.
Warehouse uses Docker and Docker Compose to automate setting up a "batteries included" development environment. The Dockerfile and :file:`docker-compose.yml` files include all the required steps for installing and configuring all the required external services of the development environment.
- Install Docker Engine
The best experience for building Warehouse on Windows 10 is to use the Windows Subsystem for Linux (WSL) in combination with both Docker for Windows and Docker for Linux. Follow the instructions for both platforms, and see Docker and Windows Subsystem for Linux Quirks for extra configuration instructions.
Verifying Docker installation
Check that Docker is installed:
Install Docker Compose
Install Docker Compose using the Docker-provided installation instructions.
Verifying Docker Compose installation
Check that Docker Compose is installed:
Verifying the neccessary ports are available
Warehouse needs access to a few local ports in order to run, namely ports
9000. You should check each of these for availability
For example, checking port
lsof -i:80 | grep LISTEN
If the port is in use, the command will produce output, and you will need to determine what is occupying the port and shut down the corresponding service. Otherwise, the port is available for Warehouse to use, and you can continue.
Building the Warehouse Container
Once you have Docker and Docker Compose installed, run:
in the repository root directory.
This will pull down all of the required docker containers, build Warehouse and run all of the needed services. The Warehouse repository will be mounted inside of the Docker container at :file:`/opt/warehouse/src/`.
Running the Warehouse container and services
You have to start the Docker services that make up the Warehouse application. These need ~4 GB of RAM dedicated to Docker to work. This is more than the default setting of the Docker Engine of 2 GB. Thus, you need to increase the memory allocated to Docker in Docker Preferences (on Mac) or Docker Settings (on Windows) by moving the slider to 4 GB in the GUI.
Then, in a terminal run the command:
This command will produce output for a while, and will not exit. While it runs, open a second terminal, and run:
This command will:
- create a new Postgres database,
- install example data to the Postgres database,
- run migrations,
- load some example data from Test PyPI, and
- index all the data for the search database.
If you get an error about xz, you may need to install the
This is highly likely on macOS and Windows.
make initdb command has finished, you are ready to continue.
Viewing Warehouse in a browser
Eventually the output of the
make serve command will cease, and you will
see a log message indicating that either the
web service has started
web_1 | [2018-05-01 20:28:14 +0000]  [INFO] Starting gunicorn 19.7.1 web_1 | [2018-05-01 20:28:14 +0000]  [INFO] Listening at: http://0.0.0.0:8000 (6) web_1 | [2018-05-01 20:28:14 +0000]  [INFO] Using worker: sync web_1 | [2018-05-01 20:28:14 +0000]  [INFO] Booting worker with pid: 15
or that the
static container has finished compiling the static assets:
static_1 | [20:28:37] Starting 'dist:compress'... static_1 | [20:28:37] Finished 'dist:compress' after 14 μs static_1 | [20:28:37] Finished 'dist' after 43 s static_1 | [20:28:37] Starting 'watch'... static_1 | [20:28:37] Finished 'watch' after 11 ms
This means that all the services are up, and web container is listening on port 80. It's accessible at http://localhost:80/.
If you are using
docker-machine on an older version of macOS or
Windows, the warehouse application might be accessible at
https://<docker-ip>:80/ instead. You can get information about the
docker container with
Logging in to Warehouse
In the development environment, the password for every account has been set to
password. You can log in as any account at
To log in as an admin user, log in as
ewdurbin with the password
password at http://localhost:80/admin/login/.
Stopping Warehouse and other services
In the terminal where
make serve is running, you can use
to gracefully stop all Docker containers, and thus the one running the
Or, from another terminal, use
make stop in the Warehouse
repository root; that'll stop all the Docker processes with
warehouse in the name.
What did we just do and what is happening behind the scenes?
The repository is exposed inside of the web container at :file:`/opt/warehouse/src/` and Warehouse will automatically reload when it detects any changes made to the code.
Running your developer environment after initial setup
You won't have to initialize the database after the first time you do
so, and you will rarely have to re-run
make build. Ordinarily, to
access your developer environment, you'll:
View Warehouse in the browser at http://localhost:80/.
Debugging the webserver
If you would like to use a debugger like pdb that allows you to drop
into a shell, you can use
make debug instead of
Errors when executing
- If the :file:`Dockerfile` is edited or new dependencies are added
(either by you or a prior pull request), a new container will need
to built. A new container can be built by running
make build. This should be done before running
make servehangs after a new build, you should stop any running containers and repeat
- To run Warehouse behind a proxy set the appropriate proxy settings in the :file:`Dockerfile`.
sqlalchemy.exec.OperationalErroris displayed in
make servehas been executed, shut down the Docker containers. When the containers have shut down, run
make servein one terminal window while running
make initdbin a separate terminal window.
"no space left on device" when using
:command:`docker-compose` may leave orphaned volumes during teardown. If you run into the message "no space left on device", try running the following command (assuming Docker >= 1.9):
docker volume rm $(docker volume ls -qf dangling=true)
This will delete orphaned volumes as well as directories that are not
(Solution found and further details available at https://github.com/chadoe/docker-cleanup-volumes)
make initdb is slow or appears to make no progress
This typically occur when Docker is not allocated enough memory to perform the
migrations. Try modifying your Docker configuration to allow more RAM for each
container, temporarily stop
make_serve and run
make initdb again.
make initdb complains about PostgreSQL Version
You built a Warehouse install some time ago and PostgreSQL has been updated.
If you do not need the data in your databases, it might be best to just blow
away your builds +
docker containers and start again:
docker volume rm $(docker volume ls -q --filter dangling=true)
Docker and Windows Subsystem for Linux Quirks
Once you have installed Docker for Windows, the Windows Subsystem for Linux, and Docker and Docker Compose in WSL, there are some extra configuration steps to deal with current quirks in WSL. Nick Janetakis has a detailed blog post on these steps, including installation, but this is a summary of the required steps:
1. In WSL, run
sudo mkdir /c and
sudo mount --bind /mnt/c /c
to mount your root drive at :file:`/c` (or whichever drive you are
using). You should clone into this mount and run
:command:`docker-compose` from within it, to ensure that when volumes
are linked into the container they can be found by Hyper-V.
2. In Windows, configure Docker to enable "Expose daemon on
tcp://localhost:2375 without TLS". Note that this may expose your
machine to certain remote code execution attacks, so use with
export DOCKER_HOST=tcp://0.0.0.0:2375 to your
:file:`.bashrc` file in WSL, and/or run it directly to enable for the
current session. Without this, the :command:`docker` command in WSL
will not be able to find the daemon running in Windows.
Styles are written in the scss variant of Sass and compiled using
:command:`gulp`. They will be automatically built when changed when
make serve is running.
Running the Interactive Shell
There is an interactive shell available in Warehouse which will automatically configure Warehouse and create a database session and make them available as variables in the interactive shell.
To run the interactive shell, simply run:
The interactive shell will have the following variables defined in it:
|db||The SQLAlchemy ORM
To use the
db object in the interactive shell, import the class you're
planning to use. For example, if I wanted to use the User object, I would
$ make shell docker-compose run --rm web python -m warehouse shell Starting warehouse_redis_1 ... ... (InteractiveConsole) >>> >>> from warehouse.accounts.models import User >>> db.query(User).filter_by(username='test').all() [User(username='test')]
You can also run the IPython shell as the interactive shell. To do so export
the environment variable WAREHOUSE_IPYTHON_SHELL prior to running the
make build step:
Now you will be able to run the
make shell command to get the IPython
Running tests and linters
PostgreSQL 9.4 is required because of
The Warehouse tests are found in the :file:`tests/` directory and are designed to be run using make.
To run all tests, in the root of the repository:
This will run the tests with the supported interpreter as well as all of the additional testing that we require.
make tests from a clean checkout of
Warehouse (namely, before trying to compile any static assets) will
fail multiple tests because the tests depend on a file
(:file:`/app/warehouse/static/dist/manifest.json`) that gets
created during deployment. So until we fix bug 1536, you'll need to
install Warehouse in a developer environment and run
before running tests; see :ref:`dev-env-install` for instructions.
If you want to run a specific test, you can use the
T=tests/unit/i18n/test_filters.py make tests
You can run linters, programs that check the code, with:
Warehouse uses black for opinionated formatting and linting. You can reformat with:
Use :command:`make` to build the documentation. For example:
The HTML documentation index can now be found at :file:`docs/_build/html/index.html`.
Building the docs requires Python 3.6. If it is not installed, the :command:`make` command will give the following error message:
make: python3.6: Command not found Makefile:53: recipe for target '.state/env/pyvenv.cfg' failed make: *** [.state/env/pyvenv.cfg] Error 127
Look at our open issues that are labelled "good first issue", find one you want to work on, comment on it to say you're working on it, then submit a pull request. Use our :doc:`submitting-patches` documentation to help.
Talk with us
Learn about Warehouse and packaging
Resources to help you learn Warehouse's context: