Repository navigation
Operator lock
The operator lock tells other people and agents that someone is changing dev or prod. Dev and prod each have their own lock, stored in their own database.
There is no on/off switch. You take it for a job, and you release it when the job is done. If you forget or crash, the lease runs out by itself.
It's on dev and prod since 2026-11-06, from release v4.9.0_110626. The commands on this page are for releases from v4.12.0_111226 on, which have the tools in scripts/ and the Django package (PR #749). Both servers run that form since 2026-11-12. For a rollback to an older release, see Rolling back past the move.
Before any write to dev or prod:
- deploys and restarts, including Neo4j
-
python -m caper.shutdown_mode onandoff - the
--applyor--executemode of a maintenance script, and backfills or migrations -
manage.pycommands that write - WAF changes (
python -m amprepo_waf edit ... --apply): the firewall fronts both servers, so take prod's lock (WAF changes) - test runs or manual testing on dev that create, edit or delete projects
You don't need it for reads, reports, dry runs, your own local database, or uploads through the site. Users are kept out by shutdown mode, which is separate.
On prod, every write is still its own explicit decision. The lock doesn't change that.
From a shell on the server. On prod, use amplicon-prod and caper.
export OPS_LOCK_HOLDER=human:<you> # an agent uses claude:<session id>
OPS="docker exec -w /srv/caper -e OPS_LOCK_HOLDER -e OPS_LOCK_TOKEN amplicon-dev /opt/venv/bin/python"
$OPS -m caper.ops_lock --expect-db caper-dev status
$OPS -m caper.ops_lock --expect-db caper-dev acquire --owner <you> --reason "what you're doing" --ttl 1h
export OPS_LOCK_TOKEN=<the token it printed>Tools run with $OPS then see that you hold it. When you're done:
$OPS -m caper.ops_lock --expect-db caper-dev releaseIf you open a new shell, set those variables again. Running acquire again as the same holder renews the lease and prints the token.
For a single command, run takes the lock, renews it while the command runs, and always releases it:
docker exec -w /srv/caper amplicon-dev /opt/venv/bin/python -m caper.ops_lock --expect-db caper-dev \
run --holder human:<you> --owner <you> --reason "backfill X" -- \
/opt/venv/bin/python ../scripts/maintenance/backfill_X.py --expect-db caper-dev --executeDon't use run for a deploy. A restart kills it inside the container, and the deploy would carry on without the lock.
Releases before v4.12.0_111226 keep ops_lock.py and shutdown_mode.py at the repository root, run from /srv. On a checkout of one of them, use the old form:
OPS="docker exec -w /srv -e OPS_LOCK_HOLDER -e OPS_LOCK_TOKEN amplicon-dev /opt/venv/bin/python"
$OPS ops_lock.py --expect-db caper-dev status # and acquire, release, wait, run
$OPS shutdown_mode.py --expect-db caper-dev status # and on, off, waitA rollback across the move uses both forms: take the lock and turn shutdown mode on the new way before git checkout, then turn it off and release the lock the old way. It's the same lock either way. The four crontab jobs (backup, disk check, cleanup, snapshot) also moved in that release, so put back the crontab from before it as well.
The default is 1 hour and the maximum is 6. Pick a length that fits the next stretch of work, and use renew if it runs long. --ttl takes values like 90s, 15m or 2h.
Tell the person you're working for, and wait:
$OPS -m caper.ops_lock --expect-db caper-dev waitbreak frees someone else's lock. Use it only when you know the holder is gone, because the lease will clear by itself anyway. break doesn't give you the lock, so run acquire afterwards.
-
Refused without the lock: the repo's write tools and
python -m caper.shutdown_mode on/off, on dev and prod.--no-lock "<reason>"overrides this for an emergency, and the reason is logged. -
Not stopped:
git checkout, restarts,dockercommands and hand-written database snippets. Nothing checks these, so they depend on you taking the lock. -
The
/admin-prepare-shutdown/page doesn't check it. Staff see a red line on/adminpages while the lock is held. If you see it, the buttons still work, so talk to the holder first. If you use the page for a deploy, take the lock from a shell first.
A database restore replaces the lock along with everything else, so take it again right after restoring.
Every acquire, renew, release, break and --no-lock is recorded in the ops_lock_log collection. The design is in docs/ops-lock-spec.md.