Skip to content

Operations

Jump to bottom
Refringe edited this page Oct 28, 2026 · 6 revisions

Operations

Running Anchor LFS day to day: what state it keeps, backups and restores, configuration changes and upgrades, signing key rotation, moving objects to S3, and removing objects nothing uses any more.

The examples use /var/lib/anchor-lfs as the data directory (the systemd layout) and the endpoint /org/repo. With Docker Compose the data directory is ./data.

What the Server Stores

Each endpoint's files live in a directory named after its path without the leading slash, with / replaced by _: the endpoint /org/repo uses org_repo.

Path Contents If it is lost
org_repo/ab/cd/<oid> in the data directory, or <s3_prefix>org_repo/ab/cd/<oid> in the bucket LFS objects, named by their SHA-256 and never changed once written Downloads of those objects fail until a clone that has them runs git lfs push --all origin
locks/org_repo/locks.json The endpoint's file locks Every lock on the endpoint is released
locks/org_repo/locks.json.lock Empty file used to serialise lock changes Nothing; it is recreated
signing.key The auto-generated HMAC key, when signing_key is not configured A new key is generated at startup, and transfer URLs issued before it fail
.tmp-* anywhere in the data directory Writes that have not finished Nothing; ones older than an hour are removed automatically

Locks and the signing key stay in the data directory with either storage backend. The configuration file, and any .env file or environment variables, are the rest of the state. They may hold the signing key and S3 credentials, so keep them as private as signing.key.

Backup

Copy the data directory, skipping temporary files:

rsync -a --exclude '.tmp-*' /var/lib/anchor-lfs/ /backup/anchor-lfs/

This is safe while the server runs: objects and signing.key never change once written, and locks.json is replaced by an atomic rename, so every copied file is complete. Repeated runs copy only new objects.

With S3 storage the same command backs up locks and the signing key. Protect the objects with your provider's bucket versioning or replication to a second bucket.

Anyone holding signing.key can create transfer URLs for every object, so store backups as securely as the server itself.

Restore

  1. Stop the server
  2. Copy the backup back: rsync -a /backup/anchor-lfs/ /var/lib/anchor-lfs/
  3. With Docker on Linux, give the files back to the container user: sudo chown -R 100:101 ./data
  4. Start the server and check curl http://localhost:5420/ready returns 200

Anything that changed after the backup was taken is lost:

  • Objects uploaded since: the batch API reports them as missing. A clone that has them uploads them again with git lfs push --all origin, which sends only the objects the server lacks.
  • Locks taken or released since: list them with git lfs locks and release stale ones with git lfs unlock --force <path> (see File Locking).
  • A different signing.key: transfer URLs issued in the last url_expiry fail once, and running the Git command again succeeds.

Configuration Changes

The configuration file and environment variables are read once, at startup. To apply a change, restart the server; it waits up to shutdown_timeout for transfers in progress.

  • systemd: sudo systemctl restart anchor-lfs
  • Docker Compose, config.toml changed: docker compose restart
  • Docker Compose, environment changed (docker-compose.yml or .env): docker compose up -d, which recreates the container with the new environment

Run anchor-lfs check before restarting (docker compose run --rm anchor-lfs check with Docker Compose): it lists every problem in the new configuration and exits 1, so a mistake never takes the server down. An invalid configuration that reaches a restart stops the server at startup with every problem logged in docker compose logs or journalctl -u anchor-lfs.

Upgrades

Release images and their tags are listed under Published Images. In production, pin a minor version such as 0.1 in docker-compose.yml rather than latest, so a pull brings fixes but no unexpected changes.

  1. Read the release notes
  2. Back up the data directory
  3. Upgrade:
    • Published image: change the tag if needed, then docker compose pull && docker compose up -d
    • Release binary: download and check the new archive, replace the binary and restart the service
    • Your own build: git fetch --tags && git checkout v0.2.0, then make docker-build && docker compose up -d, or make build and restart the service
  4. Check that curl http://localhost:5420/health reports the new version and /ready returns 200

To roll back, deploy the previous version the same way. An option added in the newer release is an unknown key to the older one, which then refuses to start, so remove such options from the configuration first. If the release notes mention a change to stored data, restore the backup taken before upgrading.

Rotating the Signing Key

Anyone holding the signing key can create transfer URLs for any object on any endpoint, so replace it if it may have leaked.

Configured key (signing_key, ANCHOR_LFS_SIGNING_KEY or ANCHOR_LFS_SIGNING_KEY_FILE):

  1. Generate a new key: openssl rand -hex 32
  2. Put it in the configuration, environment or key file in place of the old one
  3. Restart the server

Auto-generated key:

  1. Stop the server
  2. Delete signing.key from the data directory
  3. Start the server, which generates a new key

Transfer URLs signed with the old key are rejected with 403. Only those issued in the last url_expiry (default 10 minutes) were still valid, and Git LFS clients that hit one report an error and succeed when the command is run again, so rotate during a quiet period.

Presigned S3 URLs are signed with the S3 credentials, not this key. To revoke them, rotate the access key with your provider and update s3_access_key_id and s3_secret_access_key.

Moving from Local Storage to S3

The bucket uses the same layout as the data directory under s3_prefix, so copying each endpoint's directory is the whole migration. The commands use the AWS CLI; for other providers add --endpoint-url.

  1. Create the bucket and credentials (see S3-Compatible Storage)
  2. Copy each endpoint's objects, skipping temporary files: 
    aws s3 sync /var/lib/anchor-lfs/org_repo s3://my-lfs-bucket/lfs/org_repo --exclude '*.tmp-*'
  3. Set backend = "s3" and the bucket settings under [storage], then restart the server
  4. Run the same aws s3 sync again to copy objects uploaded between the first copy and the restart
  5. Check that downloads work, for example with git lfs fetch --all in a fresh clone, then delete the endpoint directories (org_repo) from the data directory

Keep the data directory itself: locks and the signing key stay there. To move from S3 back to local storage, swap the source and destination of aws s3 sync, and on Docker for Linux run sudo chown -R 100:101 ./data afterwards.

Removing Unused Objects

Anchor LFS never deletes an object, so every version of every LFS file is kept and storage only grows. With local storage, an upload is refused with 507 once the disk lacks room for it. The Git LFS API has no way to delete objects, and the server cannot tell which objects the repository's history still uses, so pruning is done by hand with a mirror of the repository.

For the endpoint /org/repo with local storage, in an empty working directory:

  1. List every object that any branch, tag or pull request uses: 
    git clone --mirror https://github.com/org/repo.git repo.git
git -C repo.git lfs ls-files --all --long | cut -d ' ' -f 1 | sort -u > keep.txt
wc -l < keep.txt
    Stop if the count is 0: every object would be deleted.
  2. List stored objects older than a day: 
    find /var/lib/anchor-lfs/org_repo -type f -mmin +1440 ! -name '.tmp-*' | sed 's|.*/||' | sort > stored.txt
  3. List the stored objects nothing uses, and review them before deleting anything: 
    comm -23 stored.txt keep.txt > prune.txt
wc -l < prune.txt
  4. Delete them (in bash or zsh): 
    while read -r oid; do rm -- "/var/lib/anchor-lfs/org_repo/${oid:0:2}/${oid:2:2}/$oid"; done < prune.txt

For S3 storage, replace step 2 with:

cutoff=$(date -u -v-1d +%Y-%m-%dT%H:%M:%S 2>/dev/null || date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%S)
aws s3api list-objects-v2 --bucket my-lfs-bucket --prefix lfs/org_repo/ \
  --query 'Contents[].[LastModified, Key]' --output text |
  awk -v cutoff="$cutoff" '$1 < cutoff { sub(/.*\//, "", $2); print $2 }' | sort > stored.txt

and step 4 with:

while read -r oid; do aws s3 rm "s3://my-lfs-bucket/lfs/org_repo/${oid:0:2}/${oid:2:2}/$oid"; done < prune.txt

Things to know before pruning:

  • The one-day cut-off protects objects from a push still in progress: Git LFS uploads objects before it pushes the commits that use them.
  • Objects used only outside the repository (unpushed branches, stashes, other remotes) are deleted too. A client that still has them uploads them again when it pushes those commits.
  • With S3 bucket versioning, deleted objects stay recoverable, and billed, until a lifecycle rule expires old versions.

Abandoned S3 Uploads

Proxied uploads over 8 MiB use S3 multipart uploads, which the server aborts when an upload fails. One cut short by a crash leaves its parts in the bucket, where they are not listed but are billed. A lifecycle rule removes them after a day:

aws s3api put-bucket-lifecycle-configuration --bucket my-lfs-bucket --lifecycle-configuration \
  '{"Rules":[{"ID":"abort-incomplete-uploads","Status":"Enabled","Filter":{"Prefix":"lfs/"},"AbortIncompleteMultipartUpload":{"DaysAfterInitiation":1}}]}'

This command replaces any lifecycle rules the bucket already has, so add the rule to them instead if there are some.

Next Steps

Clone this wiki locally