Skip to content

Deploying unreleased code to dev

Jump to bottom
Jens Luebeck edited this page Nov 11, 2026 · 5 revisions

Deploying unreleased code to the development server

This process allows a branch or individual commit to be tested on the AmpliconRepository development server without creating a tagged GitHub release.

Use this process for temporary testing on dev only. Production deployments should follow Creating a new release for AmpRepo.

When this process is appropriate

This workflow is suitable for source-only changes such as:

  • Python code, including caper/caper/views.py
  • Django templates
  • JavaScript or CSS
  • Other files mounted into the running application container

The running repository checkout is mounted directly into the Docker container. Therefore, source-only changes do not require rebuilding the Docker image.

Do not use this abbreviated process without additional planning if the change includes:

  • requirements.txt
  • Dockerfile
  • Database migrations
  • Environment or secret configuration
  • Infrastructure changes

Changes to requirements.txt or Dockerfile require rebuilding the Docker image.

What it looks like when you ignore that

Worth knowing, because the failure is confusing and the obvious fix makes it worse. Recorded from an actual occurrence on 2026-09-04.

Deploying a branch that adds a dependency and restarting gives a container that will not start. docker ps shows it Restarting, the site returns nothing at all, and logs/gunicorn.log ends in a ModuleNotFoundError for the new package, raised while Django populates INSTALLED_APPS.

Three things make the recovery non-obvious:

  1. The application's packages live in a virtualenv at /opt/venv, not in the system Python. docker exec amplicon-dev pip3 install <pkg> appears to succeed and changes nothing the application can see. Worse, the system Python has no Django, so pip resolves the dependency tree from scratch and will happily install a newer major Django alongside — on 2026-09-04 that was Django 5.2.17 next to the venv's pinned 4.0.6. The application never loaded it, but the container was left in a confusing state. Use /opt/venv/bin/pip when you mean the application's environment.

  2. You cannot docker exec into a crash-looping container. The restart policy is unless-stopped, so it cycles too fast to get a shell. To recover, stop it, check out a commit that predates the dependency, start it so it boots, install into /opt/venv, then stop, return to the test commit, and start again.

  3. --no-deps is not a shortcut. It avoids the Django-upgrade problem but installs nothing the package needs. On 2026-09-04 that produced a second crash loop on No module named 'yaml', because drf-spectacular requires PyYAML, which was not pinned in requirements.txt and was absent from the image. Prefer a constraints file, or just rebuild the image.

Which restart command does what

This matters when a container has been modified in place:

  • ~/stop-and-start-repo.sh runs docker stop then docker start. The container's writable layer survives, so anything installed into it is still there.
  • scripts/deploy/start-server.sh runs docker run, creating a new container from the image. Anything installed into the old container's writable layer is gone.

So a dev box carrying a hand-installed dependency keeps working across ordinary restarts and breaks the moment someone recreates the container — with no warning that the two commands differ. Rebuild the image instead of leaving that trap; the procedure is in Updating the Dev & Prod servers.

1. Test and push the branch

Make and test the changes locally, then commit and push the branch:

git add <changed-files>
git commit -m "Description of change"
git push -u origin <branch-name>

Record the exact commit being tested:

git rev-parse HEAD

Using an exact commit SHA makes the deployment reproducible even if additional commits are later pushed to the branch.

2. Take the operator lock and prepare the development server

Take the operator lock first, from a shell on the server. Before the move to scripts/ the commands below were different, and the deploy that brings the move uses both forms.

export OPS_LOCK_HOLDER=human:<you>     # an agent uses claude:<session id>
OPS="docker exec -w /srv/caper -e OPS_LOCK_HOLDER -e OPS_LOCK_TOKEN amplicon-dev /opt/venv/bin/python"

$OPS -m caper.ops_lock --expect-db caper-dev status    # if someone holds it, talk to them or use wait
$OPS -m caper.ops_lock --expect-db caper-dev acquire --owner <you> --reason "deploying <commit-sha>" --ttl 1h
export OPS_LOCK_TOKEN=<the token it printed>

If users may be creating or editing projects, visit /admin-prepare-shutdown/ on the development site:

  1. Confirm that no background tasks are running.
  2. Enable shutdown mode.

Or from the same shell, which refuses on if you don't hold the lock:

$OPS -m caper.shutdown_mode --expect-db caper-dev status
$OPS -m caper.shutdown_mode --expect-db caper-dev on    # refuses while tasks run
$OPS -m caper.shutdown_mode --expect-db caper-dev wait  # blocks until none run

3. Inspect the current dev checkout

Load the dev environment and enter the running checkout:

cd /home/ubuntu/AmpliconRepository-dev source caper/config.sh

Check for unexpected server-side changes:

git status --short

If tracked files have local modifications, stop and investigate them before changing revisions. Do not discard unknown changes.

Record the currently deployed commit in case a rollback is needed:

git rev-parse HEAD git log -1 --oneline

4. Check out the test commit

Fetch the latest repository state:

git fetch origin

The preferred approach is to deploy the exact commit recorded earlier:

git checkout --detach

Alternatively, deploy the latest commit from the remote branch:

git checkout --detach origin/

Confirm what is about to run:

git log -1 --oneline git status --short

5. Restart the application

Python code is loaded by Gunicorn when its workers start, so the application must be restarted:

/home/ubuntu/stop-and-start-repo.sh

A Docker image rebuild is not required for ordinary source-only changes.

During startup, the existing server process also synchronizes static files to the environment's S3 static-files location.

6. Verify the deployment

Confirm the application container is running:

docker ps --filter name=amplicon-${AMPLICON_ENV}

Inspect recent logs if necessary:

tail -n 100 logs/gunicorn.log tail -n 100 logs/stdout.txt

Smoke-test at minimum:

  • The home page
  • A CCLE project page
  • A CCLE sample page
  • The page or action affected by the change

If shutdown mode was enabled, return to /admin-prepare-shutdown/ and disable it after verification.

From a shell: $OPS -m caper.shutdown_mode --expect-db caper-dev off

Then release the lock: $OPS -m caper.ops_lock --expect-db caper-dev release

Version-display caveat

Deploying an untagged commit does not run the GitHub release workflow and therefore does not update caper/version.txt.

The site may continue displaying the most recent tagged release even though dev is running an unreleased commit. Use the following command to identify the code actually running:

cd /home/ubuntu/AmpliconRepository-dev git rev-parse HEAD git log -1 --oneline

Record the deployed commit SHA when communicating test results.

Rolling back

Hold the operator lock while rolling back. To return to the commit that was running before the test:

cd /home/ubuntu/AmpliconRepository-dev source caper/config.sh git checkout --detach /home/ubuntu/stop-and-start-repo.sh

Alternatively, restore a known tagged release:

git fetch --tags --force origin git checkout tags/ /home/ubuntu/stop-and-start-repo.sh

Verify the site again after rollback.

Promoting the change to production

After the change has been tested successfully on dev:

  1. Merge the change into main through the normal pull-request process.
  2. Create a tagged GitHub release.
  3. Follow Creating a new release for AmpRepo (https://github.com/AmpliconSuite/AmpliconRepository/wiki/Creating-a-new-release-for-AmpRepo) to deploy the release to production.

Clone this wiki locally