Skip to content

Operator lock

Jump to bottom
Jens Luebeck edited this page Nov 11, 2026 · 3 revisions

The operator lock tells other people and agents that someone is changing dev or prod. Dev and prod each have their own lock, stored in their own database.

There is no on/off switch. You take it for a job, and you release it when the job is done. If you forget or crash, the lease runs out by itself.

It's on dev and prod since 2026-11-06, from release v4.9.0_110626. The commands on this page are for a checkout that has the tools in scripts/ and the Django package (PR #749); an older checkout uses the old form.

When you need it

Before any write to dev or prod:

  • deploys and restarts, including Neo4j
  • python -m caper.shutdown_mode on and off
  • the --apply or --execute mode of a maintenance script, and backfills or migrations
  • manage.py commands that write
  • test runs or manual testing on dev that create, edit or delete projects

You don't need it for reads, reports, dry runs, your own local database, or uploads through the site. Users are kept out by shutdown mode, which is separate.

On prod, every write is still its own explicit decision. The lock doesn't change that.

How to take it

From a shell on the server. On prod, use amplicon-prod and caper.

export OPS_LOCK_HOLDER=human:<you>     # an agent uses claude:<session id>
OPS="docker exec -w /srv/caper -e OPS_LOCK_HOLDER -e OPS_LOCK_TOKEN amplicon-dev /opt/venv/bin/python"

$OPS -m caper.ops_lock --expect-db caper-dev status
$OPS -m caper.ops_lock --expect-db caper-dev acquire --owner <you> --reason "what you're doing" --ttl 1h
export OPS_LOCK_TOKEN=<the token it printed>

Tools run with $OPS then see that you hold it. When you're done:

$OPS -m caper.ops_lock --expect-db caper-dev release

If you open a new shell, set those variables again. Running acquire again as the same holder renews the lease and prints the token.

For a single command, run takes the lock, renews it while the command runs, and always releases it:

docker exec -w /srv/caper amplicon-dev /opt/venv/bin/python -m caper.ops_lock --expect-db caper-dev \
  run --holder human:<you> --owner <you> --reason "backfill X" -- \
  /opt/venv/bin/python ../scripts/maintenance/backfill_X.py --expect-db caper-dev --execute

Don't use run for a deploy. A restart kills it inside the container, and the deploy would carry on without the lock.

Checkouts from before the move to scripts/

Until PR #749, ops_lock.py and shutdown_mode.py sat at the repository root and ran from /srv. To see which form a server's checkout has:

test -f /home/ubuntu/AmpliconRepository-dev/ops_lock.py && echo "old: root" || echo "new: python -m caper.ops_lock"

The old form, for a checkout that still has the file at the root:

OPS="docker exec -w /srv -e OPS_LOCK_HOLDER -e OPS_LOCK_TOKEN amplicon-dev /opt/venv/bin/python"
$OPS ops_lock.py --expect-db caper-dev status        # and acquire, release, wait, run
$OPS shutdown_mode.py --expect-db caper-dev status   # and on, off, wait

The deploy that brings the move uses both. Take the lock and turn shutdown mode on the old way, before git checkout. After the checkout ops_lock.py is gone from the root, so turn shutdown mode off and release the lock the new way, with OPS reset to -w /srv/caper. It's the same lock either way; only the path changed. If you roll back to an older checkout, switch back to the old form.

That deploy also moves the four crontab jobs (backup, disk check, cleanup, snapshot). Edit both crontabs from scripts/README.md in the repository, under "Scheduled jobs".

The lease

The default is 1 hour and the maximum is 6. Pick a length that fits the next stretch of work, and use renew if it runs long. --ttl takes values like 90s, 15m or 2h.

If someone else holds it

Tell the person you're working for, and wait:

$OPS -m caper.ops_lock --expect-db caper-dev wait

break frees someone else's lock. Use it only when you know the holder is gone, because the lease will clear by itself anyway. break doesn't give you the lock, so run acquire afterwards.

What it does and doesn't stop

  • Refused without the lock: the repo's write tools and python -m caper.shutdown_mode on/off, on dev and prod. --no-lock "<reason>" overrides this for an emergency, and the reason is logged.
  • Not stopped: git checkout, restarts, docker commands and hand-written database snippets. Nothing checks these, so they depend on you taking the lock.
  • The /admin-prepare-shutdown/ page doesn't check it. Staff see a red line on /admin pages while the lock is held. If you see it, the buttons still work, so talk to the holder first. If you use the page for a deploy, take the lock from a shell first.

A database restore replaces the lock along with everything else, so take it again right after restoring.

Every acquire, renew, release, break and --no-lock is recorded in the ops_lock_log collection. The design is in docs/ops-lock-spec.md.

Clone this wiki locally