Project Architecture
Note: this project does not use the sections on Docker.
Poetry Setup
Using Poetry
Poetry and Python in Docker
Automated Testing
This project uses a variety of technologies to ensure smooth development and deployment processes. The core technology stack includes:
- Pytest: A testing framework that allows you to write simple as well as scalable test cases for your code.
- Pylint: A static code analysis tool that checks for errors in Python code, enforces good coding standards and practices.
- GitHub Actions: A CI/CD platform that allows you to automate tasks within your software development lifecycle.
- GitHub Workflows: Configuration files for GitHub Actions that define automated processes.
- Poetry: A dependency management tool for Python that helps you maintain a consistent and reproducible environment.
- Docker: A platform to develop, ship, and run applications inside containers.
The workflow involves creating a virtual environment, managing dependencies with Poetry, containerizing the application with Docker, and running automated tests when pull requests to main are opened.
This section summarizes how to set up poetry in projects generally and is not specific to this repo or project. Poetry simplifies the process of setting up new projects, managing dependencies, and packaging your application.
For an existing poetry project, you just need to create the virtual environment, install poetry, then install the project.
python3 -m venv venv # Create a virtual environment for the project.
source ./venv/bin/activate # Source into the virtual environment.
pip install --upgrade pip setuptools # Upgrade pip and setuptools in the virtual environment.
pip install poetry # Install poetry into the virtual environment.
poetry install # Install the project within the virtual environment.For an existing non-poetry project, you can initialize poetry and add dependencies from requirements.txt. Please ensure that your project follows the directory structure that poetry expects.
rm -r ./venv # Run this command to delete any previous virtual environment you have for this project.
python3 -m venv venv # Create a virtual environment for the project.
source ./venv/bin/activate # Source into the virtual environment.
pip install --upgrade pip setuptools # Upgrade pip and setuptools in the virtual environment.
pip install poetry # Install poetry into the virtual environment.
poetry init # Follow the steps that poetry provides to initialize the project under poetry.
poetry install # Install the project within the virtual environment.
# Now, before running this command, you'll want to remove your dev dependencies from `requirements.txt` and put them
# in a separate file, if they exist. Then, you'll want to run the below command to install all your project's release
# dependencies into poetry's management. If you do have dev dependencies, then you'll want to install them using the
# command `poetry add --group=dev dev_dependencies`. This way, dev dependencies will be ignored by `poetry` during build
# and release time, such as with `poetry build`.
cat requirements.txt | xargs poetry add # Install all the dependencies listed in `requirements.txt` into `poetry`.
poetry install # Install the project within the virtual environment.If this is a brand new project, then you can initialize your folder structure with poetry if you have it installed globally. You'll notice that once you've completed these steps, your directory structure will look like this.
python3 -m poetry new simple-api # Create the project directory with `poetry`.
cd simple-api # Change into the project directory.
git init # Initialize the Git repository.
git checkout -b main # Create the main branch and switch to it.
git remote add origin https://github.com/username/repo.git # Add your remote repository.
curl -L -o .gitignore https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore # Get the Python `.gitignore` file from GitHub.
echo "venv" >> .gitignore # Add the `venv` directory we'll make soon to .gitignore.
git add . # Add all files.
git commit -m "Initial commit" # Commit the files.
git push --set-upstream origin main # Push the main branch to the remote and set it as the upstream branch.
python3 -m venv venv # Create a virtual environment for the project.
source ./venv/bin/activate # Source into the virtual environment.
pip install --upgrade pip setuptools # Upgrade pip and setuptools in the virtual environment.
pip install poetry # Install poetry into the virtual environment.
poetry install # Install the project within the virtual environment.Two dev dependencies that you'll likely want to install into your project are pytest and pylint. pytest is a testing framework that allows you to write simple as well as scalable test cases for your code. pylint is a static code analysis tool that checks for errors in Python code, enforces PEP coding standards, and looks for code smells.
poetry add --group=dev pytest pylintPoetry expects the following directory structure for a project named simple-api. Note how the name that we gave to poetry was simple-api, but the directory it made in the simple-api folder is named simple_api. This is because python doesn't work with hyphenated package names. So it's important to note that if we choose a kebab-case project name, the actual package itself uses snake case instead.
/path/to/simple-api
├── simple_api
│ ├── __init__.py
│ ├── main.py
│ └── utils.py
├── tests
│ ├── __init__.py
│ ├── test_main.py
│ └── test_utils.py
├── venv
│ ├── bin
│ ├── include
│ ├── lib
│ ├── lib64 -> lib
│ └── pyvenv.cfg
├── LICENSE
├── poetry.lock
├── pyproject.toml
└── README.md
Notice the poetry.lock and pyproject.toml files. These should both be committed to the repo as they govern the reproducibility of the runtime environment.
To run your application or scripts with poetry, you can use the poetry run command. For example, if you want to run a script named main.py:
poetry run python simple_api/main.pyTo run tests with pytest and lint your code with pylint, you can also use the poetry run command:
poetry run pytest # Run all tests with pytest.
poetry run pylint --max-line-length=150 --disable=fixme $(git ls-files '*.py') # Lint the simple_api package with pylint.To add a dependency to your project, use:
poetry add package_nameTo add a development dependency, use:
poetry add --group=dev package_nameTo remove a dependency, use:
poetry remove package_nameTo build your project for release, use the following command:
poetry buildThis will create distribution packages in the dist directory. To publish your project to PyPI, use:
poetry publishMake sure you have configured your PyPI credentials with poetry before publishing.
To configure your PyPI credentials, create or update the ~/.pypirc file with your PyPI repository and credentials. Remember to echo .pypirc >> .gitignore.
[distutils]
index-servers =
pypi
[pypi]
repository: https://upload.pypi.org/legacy/
username: your-username
password: your-password
When running Python applications in Docker, it's not always necessary to use a virtual environment because the container itself provides isolation. However, you might need to install some packages forcefully since the system environment is designed to not have Python packages installed globally.
There is a debate about whether to install a virtual environment in a Docker container. The summary is that it can make your image larger, and the primary reason for a virtual environment is to separate program requirements from system requirements for security and compatibility. In this case, the system and the program are more tightly coupled since this is running in a stripped-down isolated Linux environment. Thus, even though most documentation advises deploying your code in a virtual environment to isolate it within your server environment, this isn't as necessary when running everything from a pre-built container. Since the container will be running, the server environment won't interfere with our Python installation.
You never want to clone repos as a step in your Dockerfile; you want to get all the files you want locally first and then move them into the container with a COPY command. The reason for this is two-fold. First is security. To quote a comment from stackoverflow:
Watch out! Docker images have a versioned file system and they remember command history. A lot of the answers will bake git credentials into your docker image. At best you are letting anyone who has the image get access to the repo until you delete the key from Github/Gitlab/etc., and at worst you are giving anyone who has the image total access to your Github/etc. account! There is almost no secure way to clone a git repo into a Dockerfile.
Secondly, as was hinted in the comment, if you put your private access token in the Dockerfile in order to clone private repos, then you are baking your credentials directly into the image. So, the alternative is to get all the files you need first and then copy them into the image. Here is a sample bash script that can be used to download and unpack repos based on their latest release:
#!/bin/bash
# Catch not enough parameters.
if [[ $# -ne 3 ]]; then
echo "Usage: `./get_releases.bash working_directory github_access_token github_owner/github_repo`"
echo "- working_directory: where all these files will be downloaded and manipulated; it's best if it's an empty directory dedicated for this purpose"
echo "- github_access_token: a long token you can get from github that you use instead of a username and password for API access"
echo "- github_owner/github_repo: of the form MyOrg/MyProj"
else
working_directory=$1
authorization="Authorization: Bearer $2" # This forms part of the HTTP header.
target_repo=$3
github_api_version='X-Github-Api-Version: 2022-11-28' # HTTP header.
target_dirname='/home/user/project/clone_for_docker
pushd "$working_directory"
# Remove previous downloads.
rm -r "$target_dirname"
# Get the tarball URL for the latest release of the repo.
api_tarball_url=$(curl -L \
-H "Accept: application/vnd.github+json" \
-H "$authorization" \
-H "$github_api_version" \
https://api.github.com/repos/$target_repo/releases/latest \
| jq -r '.tarball_url')
# Download and unpack that tarball.
curl -L \
-H "Accept: application/json" \
-H "$authorization" \
-H "$github_api_version" \
$api_tarball_url \
| tar xz
popd
fi
IMPORTANT: Do not use git clone in Docker; first clone things locally, then copy them into the image. This is for security reasons, so your credentials don't get baked into the image accidentally.
# Example Dockerfile for a simple api application deployed with gunicorn.
FROM python:3.10-alpine as base
EXPOSE 8080 # This line is documentation only and indicates that traffic to the container should be directed through port 8080.
WORKDIR /app
# Use the APK_FLAGS in case you need to install anything with apk, the Alpine Package Manager.
ARG APK_FLAGS="--no-cache --no-interactive"
ARG PIP_FLAGS="--no-cache-dir --no-input --break-system-packages"
ARG POETRY_FLAGS="--no-cache --no-interaction"
# Start a new multi-stage build.
FROM base as builder
# Just an example `apk` call to install `gcc` in the hypothetical case we needed this.
# apk add ${APK_FLAGS} gcc
# Copy your program files into the image.
COPY ./simple_api ./app
RUN pip install ${PIP_FLAGS} --upgrade pip setuptools \
&& pip install ${PIP_FLAGS} poetry \
&& POETRY_VIRTUALENVS_CREATE=false poetry build ${POETRY_FLAGS}
FROM base as final
# Copy from the builder stage.
COPY --from=builder /app/dist/*.whl .
RUN pip install ${PIP_FLAGS} gunicorn *.whl
# Command to run the application.
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "simple_api.main:app"]Automated testing in this project is set up using GitHub Actions and Workflows to ensure code quality and reliability through continuous integration.
This action sets up the Python environment and installs dependencies using poetry, allowing Workflows to run tests and other tasks.
name: Setup Python Environment
description: Checkout code, set up Python, and install dependencies.
# This is a "composite" action that can be plugged into different Workflows.
# This is used twice in the `python-tests.yaml` file for running test jobs.
runs:
using: "composite"
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
- name: Cache dependencies
id: cache-deps
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-pip-
# If you have a cache miss (to abuse the term), install dependencies.
- name: Install dependencies
if: steps.cache-deps.outputs.cache-hit != 'true'
run: |
python -m pip install --upgrade pip setuptools
python -m pip install poetry
poetry install --with dev
shell: bash
inputs:
python-version:
description: 'The Python version to use'
required: trueWorkflows for automatically running pytest and pylint on pull requests into the main branch. This ensures code quality and that all tests pass before merging.
name: Run Pytest and Pylint
# These tests only run when a pull request is opened from any branch into main.
on:
pull_request:
branches:
- main
permissions:
contents: read
jobs:
# This job executes the unit tests in the `tests/` folder on the project.
pytest:
runs-on: ubuntu-latest
steps:
- name: Setup environment
uses: ./github/actions/setup-python-env@main
with:
python-version: '3.10'
- name: Run pytest
run: |
poetry run pytest
# This job executes pylint on the project code, ensuring standardized code quality that is PEP compliant.
pylint:
runs-on: ubuntu-latest
steps:
- name: Setup environment
uses: ./github/actions/setup-python-env@main
with:
python-version: '3.10'
# Runs pylint with W0511 (fixme) disabled, which is a warning about leaving TODOs and FIXMEs in the code.
- name: Run pylint
run: |
poetry run pylint --max-line-length=150 --disable=fixme $(git ls-files '*.py')