Skip to content

Running the Server

Jump to bottom
Refringe edited this page Oct 28, 2026 · 14 revisions

Running the Server

Anchor LFS can be run directly as a binary, via Docker, or as a system service. This page covers all methods.

Quick Start

  1. Copy the example configuration: 
    cp config.toml.example config.toml
  2. Edit config.toml with your endpoints (see Configuration)
  3. Run the server: 
    make run
# or
./bin/anchor-lfs

The server listens on :5420 by default.

Running the Binary Directly

After building, run the binary from the project directory:

./bin/anchor-lfs

The server looks for config.toml in the current working directory by default. Name another file with --config, or with the ANCHOR_LFS_CONFIG environment variable (the flag wins when both are set):

./bin/anchor-lfs --config /etc/anchor-lfs/config.toml

The binary also accepts one command instead of starting the server:

Command Description
check Validates the configuration without starting the server. Prints the effective configuration (defaults and environment overrides applied, signing_key and s3_secret_access_key redacted) and exits 0, or lists every problem and exits 1. It does not contact storage.
healthcheck Checks a running server (see Health and Readiness)
version Prints the version, commit and build date (also --version)
help Lists the commands and options (also -h, --help)

--config goes before or after the command, for example anchor-lfs check --config /etc/anchor-lfs/config.toml. Any other argument prints the usage and exits with status 2.

Graceful Shutdown

The server handles SIGINT (Ctrl+C) and SIGTERM gracefully. On receiving either signal it:

  1. Stops accepting new connections
  2. Waits up to shutdown_timeout (default 30 seconds) for in-flight requests to complete, then closes any connections still open
  3. Stops the rate limiter and the authenticator cache sweepers
  4. Zeros signing key material in memory
  5. Exits with status 0, or 1 if it had to close connections at shutdown_timeout or the server failed

A second SIGINT or SIGTERM while it waits stops the server at once, cutting off any transfers still running.

An upload still running when the timeout ends is cut off, and the Git LFS client retries it. Whatever stops the server must wait longer than shutdown_timeout before killing it: stop_grace_period in Docker Compose (Docker's default is 10 seconds), --stop-timeout for docker run, TimeoutStopSec for systemd and terminationGracePeriodSeconds for Kubernetes. The examples in this wiki use 45 seconds; raise them with shutdown_timeout if your clients push large files over slow links.

Temporary Files

Uploads to local storage are written to a .tmp-* file in the endpoint's .tmp directory, and lock changes to one next to locks.json, then renamed into place once complete. A server stopped mid-upload leaves that file behind, so the server deletes .tmp-* files that have not been modified for an hour, and logs how many it removed. At startup it checks the whole data directory, and every hour after that it checks the .tmp and lock directories. A file an upload in progress is still writing is never deleted, however long the upload goes without receiving data.

Running as a Systemd Service

Create a service file at /etc/systemd/system/anchor-lfs.service:

[Unit]
Description=Anchor LFS Server
After=network.target

[Service]
Type=simple
User=anchor-lfs
Group=anchor-lfs
WorkingDirectory=/opt/anchor-lfs
ExecStart=/opt/anchor-lfs/anchor-lfs --config /etc/anchor-lfs/config.toml
Restart=on-failure
RestartSec=5
# Keep above shutdown_timeout (default 30s).
TimeoutStopSec=45

# Hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/anchor-lfs
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Then enable and start the service:

# Create a dedicated user
sudo useradd --system --no-create-home --shell /usr/sbin/nologin anchor-lfs

# Create directories
sudo mkdir -p /opt/anchor-lfs /etc/anchor-lfs /var/lib/anchor-lfs
sudo cp bin/anchor-lfs /opt/anchor-lfs/
sudo cp config.toml /etc/anchor-lfs/
sudo chown -R anchor-lfs:anchor-lfs /var/lib/anchor-lfs

# Update config to use the data directory
# Set data_directory = "/var/lib/anchor-lfs" in /etc/anchor-lfs/config.toml

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable anchor-lfs
sudo systemctl start anchor-lfs

# Check status
sudo systemctl status anchor-lfs
sudo journalctl -u anchor-lfs -f

Running with Docker

See the Docker page for Docker and Docker Compose deployment.

Health and Readiness

GET /health answers 200 whenever the process is running:

curl http://localhost:5420/health
# {"status":"ok","version":"v0.2.0"}

GET /ready also checks that the data directory accepts new files and that the storage backend is reachable (for S3, that the bucket exists and the credentials and region can reach it). It answers 200 when both pass, or 503 naming the failed check (data_directory or storage), with the cause in the server log. Results are cached for 5 seconds.

curl http://localhost:5420/ready
# {"status":"ready"}

The same checks run once at startup, so a missing bucket or wrong credentials stop the server with a checking storage error instead of failing the first request.

anchor-lfs healthcheck calls /ready on the configured listen address over loopback and exits 0 when the server is ready or 1 otherwise. It reads the config file and environment variables the same way as the server, so give it the same --config (or ANCHOR_LFS_CONFIG). The Docker image uses it as its HEALTHCHECK, which sees ANCHOR_LFS_CONFIG but not the server's --config (see Docker).

For Kubernetes, use /health as the liveness probe and /ready as the readiness probe, and allow time for a graceful shutdown:

spec:
  # Keep above shutdown_timeout (default 30s).
  terminationGracePeriodSeconds: 45
  containers:
    - name: anchor-lfs
      livenessProbe:
        httpGet:
          path: /health
          port: 5420
      readinessProbe:
        httpGet:
          path: /ready
          port: 5420

Metrics

Set metrics_address to serve Prometheus metrics at GET /metrics on a second address, separate from the Git LFS port:

[options]
metrics_address = "127.0.0.1:9420"

Metrics are off by default, and /metrics is never served on listen_address. Once the port is bound, the server logs serving metrics with its address.

Warning

The metrics endpoint has no authentication, and it shows the endpoint paths, request counts and free disk space. Bind it to loopback or to a private network that only your Prometheus server can reach.

Add the address to the Prometheus scrape configuration:

scrape_configs:
  - job_name: anchor-lfs
    static_configs:
      - targets: ["127.0.0.1:9420"]
Metric Type Labels Description
anchor_lfs_http_requests_total counter route, code Requests handled, by route and status code. Rate-limited requests have code="429"
anchor_lfs_http_request_duration_seconds histogram route Time taken to handle requests, in buckets from 5 ms to 5 minutes
anchor_lfs_http_request_body_bytes_total counter route Request body bytes read, including uploads through the server
anchor_lfs_http_response_body_bytes_total counter route Response body bytes written, including downloads through the server
anchor_lfs_storage_available_bytes gauge Free space on the filesystem that holds the data directory (local storage only)
anchor_lfs_auth_cache_requests_total counter result GitHub authentications answered from the cache (hit) or by asking GitHub (miss); present when an endpoint uses github authentication
anchor_lfs_build_info gauge version, commit Always 1, labelled with the running version and commit

The standard go_* and process_* metrics for the Go runtime and the process (memory, CPU, open files) are included too.

route is the route pattern a request matched, not its path: for example POST /org/repo/objects/batch or GET /org/repo/objects/{oid}. Each endpoint therefore has a fixed set of series, and requests to unknown paths are counted under route="/". Durations and byte counts are recorded when a request finishes, so a long transfer is counted all at once.

With S3 storage and s3_presigned_urls on (the default), clients transfer objects directly to and from the bucket, so those bytes are not in these metrics. Use the storage provider's bucket metrics for them.

Example queries:

# Requests per second by status code
sum by (code) (rate(anchor_lfs_http_requests_total[5m]))

# Download throughput through the server for /org/repo, in bytes per second
rate(anchor_lfs_http_response_body_bytes_total{route="GET /org/repo/objects/{oid}"}[5m])

# 95th percentile batch API latency for /org/repo
histogram_quantile(0.95, sum by (le) (rate(anchor_lfs_http_request_duration_seconds_bucket{route="POST /org/repo/objects/batch"}[5m])))

# Local storage runs out within a week at the last day's rate
predict_linear(anchor_lfs_storage_available_bytes[1d], 7 * 86400) < 0

Logging

By default, Anchor LFS writes JSON logs to stderr at info level, with one line per request:

{"level":"info","request_id":"a1b2c3d4e5f60718","method":"POST","path":"/org/repo/objects/batch","status":200,"duration":3.275292,"bytes":54,"ip":"203.0.113.7","time":"2026-10-20T21:27:41-04:00","message":"request"}

duration is the time taken to handle the request in milliseconds, and bytes is the size of the response body.

At startup the server logs the absolute path of the configuration file it loaded (starting anchor-lfs), its main limits and timeouts (settings), and the address it is listening on once the port is bound (listening). If the port is in use, it logs listening at fatal with the error and exits.

Requests that fail with a client error (4xx) are logged at warn and server errors (5xx) at error, so a log_level = "warn" setting keeps only failures. Successful /health and /ready checks are logged at debug, so health probes do not fill the log.

Three [options] settings control logging:

  • log_format = "console" writes human-readable lines instead of JSON (coloured only when stderr is a terminal)
  • log_level sets the lowest level logged: debug, info, warn or error
  • access_log = false turns off the per-request lines

See the Configuration page for details and the matching environment variables.

Next Steps

Clone this wiki locally