Project Architecture
Note: this project does not use the sections on Docker.
Poetry Setup
Using Poetry
Poetry and Python in Docker
Automated Testing and Publishing
Sphinx Documentation
This project uses a variety of technologies to ensure smooth development and deployment processes. The core technology stack includes:
- Pytest: A testing framework that allows you to write simple as well as scalable test cases for your code.
- Pylint: A static code analysis tool that checks for errors in Python code, enforces good coding standards and practices.
- GitHub Actions: A CI/CD platform that allows you to automate tasks within your software development lifecycle.
- GitHub Workflows: Configuration files for GitHub Actions that define automated processes.
- Poetry: A dependency management tool for Python that helps you maintain a consistent and reproducible environment.
- Docker: A platform to develop, ship, and run applications inside containers.
The workflow involves creating a virtual environment, managing dependencies with Poetry, containerizing the application with Docker, and running automated tests when pull requests to main are opened.
This section summarizes how to set up poetry in projects generally and is not specific to this repo or project. Poetry simplifies the process of setting up new projects, managing dependencies, and packaging your application.
To install poetry for the first time, please read the installation instructions here, and make sure to read the big red warning, which I somehow missed the first time. After initial installation, run the command poetry config virtualenvs.in-project true if you want your virtual environments to be created within the project folder.
After installation, you might want to symlink the installation somewhere already in your path. Something like ln -s "$HOME/poetry_env/bin/poetry" "$HOME/.local/bin/poetry".
This is a great introductory video on poetry.
For an existing poetry project, you just need to create the virtual environment, install poetry, then install the project.
poetry install # Install the project into a virtual environment.
# This drops you into a shell within the virtual environment.
# Alternatively, you can use `poetry run some_command` to execute a command within the virtual environment.
poetry shellFor an existing non-poetry project, you can initialize poetry and add dependencies from requirements.txt. Please ensure that your project follows the directory structure that poetry expects.
poetry init # Follow the steps that poetry provides to initialize the project under poetry.
# Now, before running this command, you'll want to remove your dev dependencies from `requirements.txt` and put them
# in a separate file, if they exist. Then, you'll want to run the below command to install all your project's release
# dependencies into poetry's management. If you do have dev dependencies, then you'll want to install them using the
# command `poetry add --group=dev dev_dependencies`. This way, dev dependencies will be ignored by `poetry` during build
# and release time, such as with `poetry build`.
cat requirements.txt | xargs poetry add # Install all the dependencies listed in `requirements.txt` into `poetry`.
poetry install # Install the project into a virtual environment.
# This drops you into a shell within the virtual environment.
# Alternatively, you can use `poetry run some_command` to execute a command within the virtual environment.
poetry shellIf this is a brand new project, then you can initialize your folder structure with poetry if you have it installed globally. You'll notice that once you've completed these steps, your directory structure will look like this.
poetry new simple-api # Create the project directory with `poetry`.
cd simple-api # Change into the project directory.
git init # Initialize the Git repository.
git checkout -b main # Create the main branch and switch to it.
git remote add origin https://github.com/username/repo.git # Add your remote repository.
curl -L -o .gitignore https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore # Get the Python `.gitignore` file from GitHub.
echo ".venv" >> .gitignore # Add the `.venv` directory we'll make soon to .gitignore.
git add . # Add all files.
git commit -m "Initial commit" # Commit the files.
git push --set-upstream origin main # Push the main branch to the remote and set it as the upstream branch.
poetry install # Install the project within the virtual environment.
# This drops you into a shell within the virtual environment.
# Alternatively, you can use `poetry run some_command` to execute a command within the virtual environment.
poetry shellTwo dev dependencies that you'll likely want to install into your project are pytest and pylint. pytest is a testing framework that allows you to write simple as well as scalable test cases for your code. pylint is a static code analysis tool that checks for errors in Python code, enforces PEP coding standards, and looks for code smells.
poetry add --group=dev pytest pylintPoetry expects the following directory structure for a project named simple-api. Note how the name that we gave to poetry was simple-api, but the directory it made in the simple-api folder is named simple_api. This is because python doesn't work with hyphenated package names. So it's important to note that if we choose a kebab-case project name, the actual package itself uses snake case instead.
/path/to/simple-api
├── simple_api
│ ├── __init__.py
│ ├── main.py
│ └── utils.py
├── tests
│ ├── __init__.py
│ ├── test_main.py
│ └── test_utils.py
├── venv
│ ├── bin
│ ├── include
│ ├── lib
│ ├── lib64 -> lib
│ └── pyvenv.cfg
├── LICENSE
├── poetry.lock
├── pyproject.toml
└── README.md
Notice the poetry.lock and pyproject.toml files. These should both be committed to the repo as they govern the reproducibility of the runtime environment.
To run your applications or scripts, first get into the virtual environment that poetry made using the poetry shell command. Then you can just run things as normal.
poetry shell
python simple_api/main.pyRemember that in order to run tests, you need to use the poetry install --with=dev command.
pytest is the testing framework discussed here. To run the tests, simply type in pytest. Similarily, pyright is used to check the correctness of the code, and is likewise ran by simply typing in pyright.
To run pylint to lint the code, first create a .pylintrc file. Here is a sample file:
[FORMAT]
# Set maximum line length
max-line-length=150
[MESSAGES CONTROL]
# Disable specific Pylint warnings and errors; these are comma-separated.
disable=
trailing-whitespace,
Then, you can simply run pylint $(git ls-files '*.py').
To add a dependency to your project, use:
poetry add package_nameTo add a development dependency, use:
poetry add --group=dev package_nameTo remove a dependency, use:
poetry remove package_nameWhen running Python applications in Docker, it's not always necessary to use a virtual environment because the container itself provides isolation. However, you might need to install some packages forcefully since the system environment is designed to not have Python packages installed globally.
There is a debate about whether to install a virtual environment in a Docker container. The summary is that it can make your image larger, and the primary reason for a virtual environment is to separate program requirements from system requirements for security and compatibility. In this case, the system and the program are more tightly coupled since this is running in a stripped-down isolated Linux environment. Thus, even though most documentation advises deploying your code in a virtual environment to isolate it within your server environment, this isn't as necessary when running everything from a pre-built container. Since the container will be running, the server environment won't interfere with our Python installation.
You never want to clone repos as a step in your Dockerfile; you want to get all the files you want locally first and then move them into the container with a COPY command. The reason for this is two-fold. First is security. To quote a comment from stackoverflow:
Watch out! Docker images have a versioned file system and they remember command history. A lot of the answers will bake git credentials into your docker image. At best you are letting anyone who has the image get access to the repo until you delete the key from Github/Gitlab/etc., and at worst you are giving anyone who has the image total access to your Github/etc. account! There is almost no secure way to clone a git repo into a Dockerfile.
Secondly, as was hinted in the comment, if you put your private access token in the Dockerfile in order to clone private repos, then you are baking your credentials directly into the image. So, the alternative is to get all the files you need first and then copy them into the image. Here is a sample bash script that can be used to download and unpack repos based on their latest release:
#!/bin/bash
# Catch not enough parameters.
if [[ $# -ne 3 ]]; then
echo "Usage: `./get_releases.bash working_directory github_access_token github_owner/github_repo`"
echo "- working_directory: where all these files will be downloaded and manipulated; it's best if it's an empty directory dedicated for this purpose"
echo "- github_access_token: a long token you can get from github that you use instead of a username and password for API access"
echo "- github_owner/github_repo: of the form MyOrg/MyProj"
else
working_directory=$1
authorization="Authorization: Bearer $2" # This forms part of the HTTP header.
target_repo=$3
github_api_version='X-Github-Api-Version: 2022-11-28' # HTTP header.
target_dirname='/home/user/project/clone_for_docker
pushd "$working_directory"
# Remove previous downloads.
rm -r "$target_dirname"
# Get the tarball URL for the latest release of the repo.
api_tarball_url=$(curl -L \
-H "Accept: application/vnd.github+json" \
-H "$authorization" \
-H "$github_api_version" \
https://api.github.com/repos/$target_repo/releases/latest \
| jq -r '.tarball_url')
# Download and unpack that tarball.
curl -L \
-H "Accept: application/json" \
-H "$authorization" \
-H "$github_api_version" \
$api_tarball_url \
| tar xz
popd
fi
IMPORTANT: Do not use git clone in Docker; first clone things locally, then copy them into the image. This is for security reasons, so your credentials don't get baked into the image accidentally.
# Example Dockerfile for a simple api application deployed with gunicorn.
FROM python:3.10-alpine as base
EXPOSE 8080 # This line is documentation only and indicates that traffic to the container should be directed through port 8080.
WORKDIR /app
# Use the APK_FLAGS in case you need to install anything with apk, the Alpine Package Manager.
ARG APK_FLAGS="--no-cache --no-interactive"
ARG PIP_FLAGS="--no-cache-dir --no-input --break-system-packages"
ARG POETRY_FLAGS="--no-cache --no-interaction"
# Start a new multi-stage build.
FROM base as builder
# Just an example `apk` call to install `gcc` in the hypothetical case we needed this.
# apk add ${APK_FLAGS} gcc
# Copy your program files into the image.
COPY ./simple_api ./app
RUN pip install ${PIP_FLAGS} --upgrade pip setuptools \
&& pip install ${PIP_FLAGS} poetry \
&& POETRY_VIRTUALENVS_CREATE=false poetry build ${POETRY_FLAGS}
FROM base as final
# Copy from the builder stage.
COPY --from=builder /app/dist/*.whl .
RUN pip install ${PIP_FLAGS} gunicorn *.whl
# Command to run the application.
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "simple_api.main:app"]Automated testing and publishing is done via github workflows.
The following file, which I name ci-cd.yaml (and should live in $REPO_HOME/.github/workflows), does a few things worth discussing. First, it runs three separate testing frameworks in parallel: pytest (for unit testing), pylint (for ensuring code quality standards), and pyright (for ensuring no sneaky bugs get introduced). These tests are set to run whenever a pull request is made or whenever a new release is published.
There is a commented out section below these tests for running the publish job. If you plan on publishing the package to pypi, then uncomment out that section of the workflow file. By uncommenting that code, you make it so that when the project is released, it will first run the three tests as mentioned above, and only if all those tests pass will it then execute the publishing code to build and publish the repo (via poetry) to pypi. To have this work, you must first set up your pypi token in the github repo. In github, go to settings -> Secrets and Variables -> Actions, and add a new repository secret called PYPI_TOKEN, which is the token provided to you for authentication by pypi.
name: Run Tests and Publish on Release
on:
pull_request:
branches:
- main
release:
types:
- published
permissions:
contents: read
jobs:
# Pytest job
pytest:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
- name: Install Poetry
uses: snok/install-poetry@v1
- name: Install dependencies
run: |
poetry install --with dev
- name: Run pytest
run: |
poetry run pytest
# Pylint job
pylint:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
- name: Install Poetry
uses: snok/install-poetry@v1
- name: Install dependencies
run: |
poetry install --with dev
- name: Run pylint
run: |
poetry run pylint $(git ls-files '*.py')
# Pyright job
pyright:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.12'
- name: Install Poetry
uses: snok/install-poetry@v1
- name: Install dependencies
run: |
poetry install --with dev
- name: Run pyright
run: |
poetry run pyright
# Publish job (only runs on release)
# publish:
# if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
# needs: [pytest, pylint, pyright]
# runs-on: ubuntu-latest
# steps:
# - name: Checkout code
# uses: actions/checkout@v2
# - name: Set up Python
# uses: actions/setup-python@v4
# with:
# python-version: '3.12'
# - name: Install Poetry
# uses: snok/install-poetry@v1
# - name: Install dependencies
# run: |
# poetry install --with dev
# - name: Build the package
# run: |
# poetry build
# - name: Publish to PyPI
# env:
# PUBLISHING_TOKEN: ${{ secrets.PYPI_TOKEN }}
# run: |
# poetry publish --username __token__ --password $PUBLISHING_TOKENSphinx is a tool used to generate professional looking documentation automatically from the content of docstrings. See this style guide on using the Numpy docstring format. This allows you to document the project in the code itself.
- Install Sphinx with the ReadTheDocs (rtd) theme via the command
poetry add --group=dev sphinx sphinx-rtd-theme. - Set Sphinx up for the project via the command
sphinx-quickstart docs. This sets up Sphinx in thedocs/folder, and creates one if it doesn't exist. - Navigate to
docs/conf.pyand add the following import statement:
import os
import sys
sys.path.insert(0, os.path.abspath('../project_name')) # Adjust the path as necessary.This makes your code available to the viewcode extension.
- Set up your extensions. You'll see a plain
extensions = []line; replace it with the following:
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.napoleon',
'sphinx.ext.viewcode',
'sphinx.ext.githubpages'
]autodoc generates documentation automatically from your docstrings. napoleon interprets the numpy style docstrings. viewcode lets you actually inspect the source code from the documentation. githubpages sets things up for deployment of the documentation to GitHub Pages.
-
Set the theme by replacing the line
html_theme = 'alabaster'withhtml_theme = 'sphinx_rtd_theme'. -
Set-up napoleon to interpret Numpy-style docstrings. You can just add this to the bottom of the file:
napoleon_google_docstring = False
napoleon_numpy_docstring = True- Next, generate our
.rstfiles for the project with the commandsphinx-apidoc -o docs/ my_project/.
Edit the index.rst file. You'll notice firstly that there is also a folder called source/ that holds .rst files for your project. You can include these in the index.rst file by specifying them directly. This file is also the homepage for documentation, so you can edit its contents as you wish for what you want your documentation's landing page to look like. Here's an example file:
my_project documentation
=========================
This is the documentation for the `my_project` python package found here: https://github.com/OrgName/my-project.
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
source/my_project
To generate the documentation, run sphinx-build -b html docs docs/_build/html.
Now, we can setup an orphan branch called gh-pages that will automatically be read by GitHub and set up the GitHub Pages website for you automatically from this repo. This means that we create an orphan repo, then delete all of its contents (except for the .venv and .git folders), and push this empty repo to origin. Then, we use git worktree to add this orphan repo to track the docs/_build/html folder. This means that when we cd into docs/_build/html, the branch that is being tracked will be the gh-pages branch, not our current branch. But, when we cd out it, we can see that we're back to our working branch being tracked. You can see this by examining the input of git branch after setup.
What this does is allows us to generate the documentation within our working branch, cd into the docs/_build/html folder, and then push all those files to the gh-pages repo, which will then automatically publish the documentation of the application.
git checkout --orphan gh-pages # Create the orphan branch in the project. This is a branch detached from `main`.
rm -r * && rm .* # Removes all contents of the repo except for hidden folders (namely `.git` and `.venv`).
# Commit empty repo to project. You should see that GitHub automatically created the GitHub Pages for you
# This page should 404 since the repo is empty once you push to origin.
git commit --allow-empty -m "Initial empty documentation commit"
git push origin gh-pages # Set gh-pages to remote tracking.
git checkout main # Checkout main again.
rm -r ./docs/_build/html # Remove the `html` folder generated by Sphinx.
git worktree add ./docs/_build/html gh-pages # Add `gh-pages` to track a new folder called `html` in `docs/_build`.
sphinx-build -b html docs docs/_build/html # Rebuild the documentation into the newly tracked folder.
cd docs/_build/html # `cd` into the documentation directory for `gh-pages` branch.
git branch # Confirm that `gh-pages` is being tracked.
git add --all # Add all files in this directory as it represents a website essentially.
git commit -m "Add initial application documentation to the repo" # Add the documentation to the repo.
# Push to the `gh-pages` repo.
# You should now have your documentation live on your GitHub Pages page, once tests complete.
git push You should also commit the rest of the Sphinx files to main (i.e. .rst files, conf.py, etc.).
