Config TLS

This page covers TLS served by ALPS itself, through the [tls], [tls.letsencrypt] and [tls.letsencrypt.s3] sections. It explains how to serve certificate files you provide, how ALPS gets and renews Let's Encrypt certificates, how to test against the staging directory, how several nodes share certificates through S3, and what to set when another server holds port 80. Behind a reverse proxy that terminates TLS, leave all of it off and follow Setup-Behind-a-Reverse-Proxy. The full reference is section 6 of docs/CONFIGURATION.md, and docs/TLS.md has the details.

Choosing where TLS ends

Setup	`[tls]`	Who looks after the certificate
A reverse proxy terminates TLS	`enabled = false` (the default)	The proxy. ALPS speaks plain HTTP, best on `127.0.0.1`. See Setup-Behind-a-Reverse-Proxy.
ALPS serves HTTPS with certificates you provide	`enabled = true`, `provider = "file"`	You. An outside tool renews the files, and ALPS is restarted after each renewal.
ALPS serves HTTPS with Let's Encrypt	`enabled = true`, `provider = "letsencrypt"`	ALPS, which orders, stores and renews them. See Setup-Lets-Encrypt-Standalone.

With enabled = true:

[server] addr (default ":1323") becomes the HTTPS listener. For browsers to reach it without a port, set addr = ":443", or forward port 443 to it. Ports below 1024 need root or CAP_NET_BIND_SERVICE, which the shipped dist/alps.service grants.
ALPS accepts TLS 1.2 and 1.3 only. A client limited to TLS 1.1 gets a protocol_version alert. ALPS offers HTTP/2 and HTTP/1.1.
Plain HTTP sent to the HTTPS port gets 400 Client sent an HTTP request to an HTTPS server. ALPS has no HTTP-to-HTTPS redirect. With "file", nothing listens on port 80. With "letsencrypt", the listener on acme_http_addr answers ACME challenges only and returns 404 for every other path.
Session and login cookies get the Secure flag, and the origin check expects https:// and the host the browser used.
Set [webauthn] rpid and origins to the browser-facing hostname, with the port when it is not 443, or security keys do not work. See Config-Rate-Limiting-and-WebAuthn.

ALPS reads these sections once, at startup. Restart it to apply a change: systemctl reload alps sends SIGHUP, which only reopens the log file.

`[tls]`

Key	Default	Meaning
`enabled`	`false`	Turns on HTTPS in ALPS. While it is `false`, ALPS ignores the other keys of `[tls]`, `[tls.letsencrypt]` and `[tls.letsencrypt.s3]`, so even an invalid `provider` there does not stop startup. The file must still be valid TOML, with each key of the right type.
`provider`	none	Exactly `"file"` or `"letsencrypt"`, in lower case. Anything else, or no value, stops startup.
`cert_file`	none	With `"file"`: the PEM file with the server certificate and its intermediate certificates.
`key_file`	none	With `"file"`: the PEM file with the matching private key.

Certificate files (`provider = "file"`)

[server]
addr = ":443"

[tls]
enabled = true
provider = "file"
cert_file = "/etc/alps/tls/fullchain.pem"
key_file = "/etc/alps/tls/privkey.pem"

cert_file holds the server certificate followed by the intermediate certificates, as in a "fullchain" file. ALPS sends the chain exactly as the file has it.
key_file holds the private key that matches the certificate. It must not be protected by a passphrase.
ALPS serves this one certificate to every client, whatever name the client asks for, and to clients that ask for none. To serve several hostnames, use a certificate that lists all of them.
The user ALPS runs as must be able to read both files. The shipped systemd unit hides /home (ProtectHome=true), so keep the files elsewhere, for example under /etc/alps.

Renewing the files

ALPS loads the pair once, at startup, and keeps serving it until it stops. After the files are replaced, restart ALPS (systemctl restart alps), for example from the deploy hook of the tool that renews them. A SIGHUP (systemctl reload alps) leaves the old certificate in place.

Errors that stop startup

Each of these ends ALPS with exit status 1:

Log message	Cause
`Failed to build TLS config: unsupported tls provider:`	`enabled = true` with no `provider`.
`Failed to build TLS config: unsupported tls provider: File`	A `provider` that is not exactly `"file"` or `"letsencrypt"` (here, upper case).
`Failed to initialize TLS manager: cert_file and key_file required`	One of the two keys is missing.
`Failed to initialize TLS manager: load certificate: open /etc/alps/tls/fullchain.pem: no such file or directory`	A path that does not exist. A file the ALPS user may not read ends the line with `permission denied` instead.
`Failed to initialize TLS manager: load certificate: tls: private key does not match public key`	The key belongs to another certificate.
`Failed to initialize TLS manager: load certificate: tls: failed to parse private key`	The key is encrypted with a passphrase.
`Failed to initialize TLS manager: load certificate: tls: failed to find certificate PEM data in certificate input, but did find a private key; PEM inputs may have been switched`	`cert_file` and `key_file` are swapped.

Let's Encrypt (`provider = "letsencrypt"`)

ALPS orders a certificate for each name in domains, stores it, serves it and renews it. By default it uses the Let's Encrypt production directory. When it registers the ACME account, it accepts the CA's terms of service for you.

`[tls.letsencrypt]`

Key	Default	Meaning
`email`	none (required)	The contact address of the ACME account.
`domains`	none (required)	Every hostname browsers use. At least one is needed. The list is exact: no wildcards, and each name gets a certificate of its own.
`default_domain`	the first entry of `domains`	The certificate for clients that send no server name (SNI). ALPS does not check the value at startup. Set it to one of `domains`: any other name makes every handshake without a server name fail.
`storage_provider`	`"s3"`	`"s3"`: certificates are kept in the `[tls.letsencrypt.s3]` bucket, with a local copy in `cache_dir`. `"file"`: `cache_dir` only, for a single node. Because the default is `"s3"`, a single node without a bucket must set `"file"`. With `[cluster] enabled = true` it must be `"s3"`.
`cache_dir`	`"cert-cache"`	The local certificate directory. A relative path is relative to the process's working directory, so use an absolute one.
`sync_interval_minutes`	`5`	With `"s3"`: how often a node retries uploading certificates it could only store locally while S3 was failing. 0 or a negative value means 5.
`acme_http_addr`	`":80"`	Where the HTTP-01 challenge listener runs.
`directory_url`	`""` (Let's Encrypt production)	Another ACME directory, such as Let's Encrypt staging. See Staging and other directories.
`warm_rsa_certs`	`false`	Also order an RSA certificate for each name ahead of need, beside the ECDSA one.

A single node

[server]
addr = ":443"

[tls]
enabled = true
provider = "letsencrypt"

[tls.letsencrypt]
email = "admin@example.com"
domains = ["webmail.example.com"]
storage_provider = "file"               # the default is "s3"
cache_dir = "/var/lib/alps/cert-cache"
# acme_http_addr = ":80"                # the default

The shipped systemd unit sets no WorkingDirectory and makes the file system read-only (ProtectSystem=strict), so the default cache_dir would be /cert-cache, which cannot be written. Create the directory's parent and allow writes there in a drop-in:

install -d -o alps -g alps -m 700 /var/lib/alps
systemctl edit alps

[Service]
ReadWritePaths=/var/lib/alps

Then run systemctl restart alps. With Let's Encrypt, a shutdown also stops the certificate workers, so it takes up to 40 seconds with "file" storage, up to 50 with S3 and up to 55 in a cluster. The shipped unit's TimeoutStopSec=60 covers all of these. See docs/OPERATIONS.md.

Challenges and ports

Let's Encrypt checks that you control a name before it issues a certificate. ALPS answers two kinds of checks:

TLS-ALPN-01, tried first, on the HTTPS listener ([server] addr). It must be reachable from the internet as port 443.
HTTP-01, tried next, on acme_http_addr. It must be reachable as port 80.

If port 443 does not reach ALPS, each order spends a failed TLS-ALPN-01 attempt and then succeeds over HTTP-01. If neither port reaches ALPS, the order fails, and ALPS logs for example (with format = "console"):

level=ERROR msg="TLS: failed to get certificate" server_name=webmail.example.com error="acme/autocert: unable to satisfy \"https://...\" for domain \"webmail.example.com\": no viable challenge type found"

The HTTP-01 listener:

runs on every node that uses Let's Encrypt, and cannot be turned off;
answers only paths under /.well-known/acme-challenge/, and returns 404 for everything else, with no redirect;
answers 403 unless the Host header is exactly one of domains, in lower case and with no port;
uses the [server] read, write and idle timeouts, and at shutdown stops with the HTTPS listener, within the same 30 seconds;
does not stop ALPS when it cannot bind, because the port is taken or because ALPS may not bind ports below 1024. ALPS then logs a line such as ACME HTTP-01 handler error: listen tcp 127.0.0.1:80: bind: address already in use and runs without the listener, so only TLS-ALPN-01 can succeed.

When certificates are ordered

Warm passes. A single node runs them itself. In a cluster only the leader does. The first pass runs 60 to 120 seconds after startup, then one every 12 hours, never more often than every 15 minutes, with 2 seconds between names. A pass orders an ECDSA certificate for every name in domains that has none, and an RSA one as well with warm_rsa_certs = true. A new node therefore has its certificates one to two minutes after it starts, plus the time the orders take, without any visitor.
On demand. A handshake for a listed name that has no certificate yet starts an order, and the handshake waits for it. Once its ClientHello names a listed domain, that handshake has two minutes, apart from [server] read_timeout_sec, for the certificate to be found or ordered and for the rest of the handshake, so the first visit can wait out a slow order. The ClientHello itself, and the HTTP request after the handshake, keep the server's usual timeouts. An order that takes longer than two minutes fails that first handshake. Waiting for the warm pass after adding a name spares the first visitors the wait.
RSA. Clients without ECDSA support get an RSA certificate. Without warm_rsa_certs, it is ordered at the first such handshake.
Renewal. A certificate is due for renewal when it has min(30 days, one third of its lifetime) left, or up to an hour later. A renewal that fails is retried 30 to 60 minutes later, and the current certificate is served meanwhile, as long as it is valid. ALPS does not serve a certificate that has expired, or that is not valid yet: the handshake fails, and ALPS logs TLS: cached certificate is not valid at error level, with server_name and the reason.
Status. Each warm pass logs the state of every certificate it checks, with domain, flavor, not_after and remaining: cert warmer: certificate is valid at info level, cert warmer: certificate is due for renewal as a warning once it is inside its renewal window, and cert warmer: certificate has expired or cert warmer: certificate is not yet valid as errors. A renewal that keeps failing therefore shows as a warning every 12 hours, before the certificate expires.

Names and SNI

A handshake for a name that is not in domains fails with a TLS alert and gets no certificate. ALPS logs nothing about it unless [server] debug = true, which logs http: TLS handshake error from 192.0.2.10:53433: host not configured: other.example.com at debug level (not shown when [logging] level is set higher). debug is for development only, so keep it off in production.
Server names in handshakes match without regard to case.
A client that sends no server name gets the default_domain certificate. That includes a browser opening https://203.0.113.10/, since a client does not send an IP address as the server name. The certificate does not match the address, so the browser warns.
A default_domain that is not in domains makes every handshake without a server name fail. ALPS starts with such a value without a warning, so check it against domains.

What ALPS stores

cache_dir is created with mode 0700, and its files with mode 0600. It holds:

File	Content
`acme_account+key`	The ACME account key.
`webmail.example.com`	The ECDSA certificate for that name, with its private key.
`webmail.example.com+rsa`	The RSA certificate, when one was ordered.

With storage_provider = "file", the ACME challenge tokens also pass through it while an order runs, and are removed once used. With "s3" they stay out of it (see What happens). With "file", this directory is the only copy. Keep it across restarts and reinstalls, and back it up like any private key. If it is lost, ALPS orders every certificate again.

Checking that it works

The certificate manager's lines go through [logging] like every other line: to its output, in its format, and only when their level reaches its level. Under systemd with the default output, they are in journalctl -u alps. A good start looks like this, with format = "console" (time fields left out):

level=INFO msg="using file-based certificate cache" cache_dir=/var/lib/alps/cert-cache
level=INFO msg="TLS running in single-instance mode (no cluster leader election)"
level=INFO msg="TLS manager initialized" domains=[webmail.example.com] email=admin@example.com storage=file default_domain=webmail.example.com
level=INFO msg="Starting ACME HTTP-01 challenge handler on :80"
level=INFO msg="Starting server on :443"

The last two lines can come in either order. One to two minutes later, the first warm pass logs a line for each certificate, such as:

level=INFO msg="cert warmer: certificate is valid" domain=webmail.example.com flavor=ecdsa not_after=... remaining=...

Failed orders show up at error level as:

TLS: failed to get certificate, with server_name and error, for an order a handshake started;
cert warmer: failed to obtain certificate, with domain, flavor=ecdsa and error, for an order a warm pass started.

Search the log for cert warmer: certificate is due for renewal and cert warmer: certificate has expired too.

To see the certificate ALPS serves:

openssl s_client -connect webmail.example.com:443 -servername webmail.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -dates -ext subjectAltName

Errors that stop startup

Log message	Cause
`Failed to initialize TLS manager: email required`	No `email`.
`Failed to initialize TLS manager: at least one domain required`	No `domains`, or an empty list.
`Failed to initialize TLS manager: unsupported storage provider: disk`	A `storage_provider` other than `"s3"` or `"file"`.
`Failed to initialize TLS manager: S3 bucket validation failed after 5 attempts: ...`, or `... S3 bucket validation cancelled after <n> attempts: context deadline exceeded`, followed by the last attempt's error	The bucket could not be reached, in five attempts within 30 seconds. The last error says why, for example a missing bucket, refused credentials or an endpoint that does not answer. This is also what a single node gets when it leaves `storage_provider` at its default `"s3"` without configuring a bucket.
`TLS letsencrypt storage_provider must be "s3" when clustering is enabled (got "file"): ...`	`[cluster] enabled = true` with `"file"` storage.

Staging and other directories

While you set things up, order from the Let's Encrypt staging directory, so that failed attempts do not count against the production limits:

[tls.letsencrypt]
directory_url = "https://acme-staging-v02.api.letsencrypt.org/directory"

Browsers do not trust staging certificates, so use staging only to check that ordering works.

Any directory_url other than the production one gets a store of its own. ALPS appends -acme-<8 hex digits> to cache_dir, and keeps the S3 objects in a folder acme-<8 hex digits>/ under the prefix. The digits come from the URL. For the staging URL above they are 67f27506, so cache_dir = "/var/lib/alps/cert-cache" becomes /var/lib/alps/cert-cache-acme-67f27506. At startup ALPS logs using non-default ACME directory - certificate store namespaced to keep it isolated from production at warn level, with the directory, the cache_dir and the S3 prefix it uses.
With prefix = "alps/", the staging objects are named like alps/acme-67f27506/webmail.example.com, apart from the production objects. Without a prefix, they go into acme-67f27506/ at the bucket root.
To move to production, remove directory_url (or set it to "") and restart. ALPS then uses the production store, which staging never wrote to, and orders production certificates at the next warm pass or the first handshake. Delete the -acme-67f27506 directory, and the staging objects in S3, once you no longer need them.
ALPS has no setting for external account binding, so a CA that requires it cannot be used.

Sharing certificates through S3

With storage_provider = "s3", the certificates live in an S3 bucket (AWS S3, or an S3-compatible store such as MinIO), and cache_dir keeps a local copy on each node. Several nodes using Let's Encrypt need this storage, and they also need [cluster] enabled = true. In a cluster only the leader runs the warm passes and may store certificates, so the certificates in S3 are the ones the leader ordered and renewed, and the other nodes read them from there. Without [cluster], each node orders on its own, and several nodes can order the same certificate at about the same time. The cluster itself is covered on Config-Cluster, Setup-Cluster and Clustering.

`[tls.letsencrypt.s3]`

Key	Default	Meaning
`endpoint`	none (AWS S3)	The endpoint of an S3-compatible store, for example `"https://s3.example.com:9000"`. Without a scheme, ALPS adds `https://`. With an endpoint, ALPS uses path-style addressing. Leave it unset for AWS S3.
`bucket`	none	The bucket name. ALPS checks it at startup, within 30 seconds, and does not start if it cannot reach it.
`access_key`, `secret_key`	none	Used only when both are set. Otherwise the AWS SDK's default credentials apply: its environment variables, the shared configuration files, or an instance role.
`region`	`us-east-1` with an `endpoint`	Without an `endpoint`, the AWS SDK works out the region from its own configuration.
`prefix`	`""` (bucket root)	A folder for the objects. `"alps"` and `"alps/"` both put them under `alps/`. Without a prefix, each upload logs `S3Cache.Put: no prefix configured - storing in bucket root` as a warning.

[tls.letsencrypt]
email = "admin@example.com"
domains = ["webmail.example.com"]
# storage_provider = "s3"                  # the default
cache_dir = "/var/lib/alps/cert-cache"

[tls.letsencrypt.s3]
endpoint = "https://s3.example.com:9000"   # leave out for AWS S3
bucket = "alps-certs"
region = "us-east-1"
prefix = "alps/"
# Omit both keys to use the AWS default credentials (environment, instance role).
# access_key = "..."
# secret_key = "..."

What happens

Startup. ALPS checks the bucket with up to 5 HeadBucket requests, with pauses of 1, 2, 4 and 8 seconds between them, all within 30 seconds, and exits if none succeeds. A good start logs using custom S3 endpoint endpoint=https://s3.example.com:9000 (with an endpoint), validating S3 bucket access bucket=alps-certs, using hybrid file+S3 certificate cache with periodic sync local_cache_dir=/var/lib/alps/cert-cache s3_bucket=alps-certs s3_prefix=alps/ sync_interval=5m0s, and certificate sync worker: startup sync complete.
Object names. The objects are the prefix followed by the file names in cache_dir: alps/acme_account+key, alps/webmail.example.com, alps/webmail.example.com+rsa.
Challenge tokens. These are written to S3 only, for the length of a validation, and deleted afterwards. They never go to cache_dir. That way any node can answer Let's Encrypt's request, wherever the load balancer sends it.
Reads. A node reads a certificate from cache_dir first, and from S3 when it has no local copy. It then keeps the S3 copy locally. How the nodes pick up a renewed certificate is described in docs/TLS.md and docs/CLUSTER.md.
Writes and S3 outages. A new certificate goes to S3 first and then to cache_dir. If S3 fails, the certificate is kept in cache_dir only, and the node leaves S3 alone for 30 seconds. It tries the upload once more after those 30 seconds, then every sync_interval_minutes (5 when it is 0 or negative) while anything is still waiting, and after each warm pass. A check with nothing waiting does nothing. While S3 is failing, warm passes are put off.
Access. The bucket holds every certificate's private key and the ACME account key. Give access to the ALPS nodes only. ALPS sends a HeadBucket request for the bucket at startup, and GetObject, PutObject and DeleteObject requests for objects under the prefix. It never lists the bucket. Prefer an instance role or the environment over keys in config.toml, and when the keys are in the file, keep it readable only by root and the ALPS user.

ACME through another server on port 80

This setup applies when ALPS serves HTTPS with Let's Encrypt and another web server holds port 80, for example to redirect plain-HTTP visitors to HTTPS. That server forwards /.well-known/acme-challenge/ to ALPS's acme_http_addr, and handles everything else itself.

[server]
addr = ":443"

[tls.letsencrypt]
acme_http_addr = "127.0.0.1:8080"

server {
    listen 80;
    server_name webmail.example.com;

    location /.well-known/acme-challenge/ {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;   # the name, without a port
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

ALPS answers 403 to a challenge request whose Host carries a port. Nginx's $host drops the port. $http_host keeps whatever the client sent, port included.
Port 443 must still reach ALPS, because ALPS terminates TLS and answers TLS-ALPN-01 there. Prefer letting ALPS listen on it itself (addr = ":443", as above), or a port forward that keeps the client's address. Use a TCP passthrough only when another service must share port 443, and read the next point first. A passthrough in HAProxy, with ALPS on addr = "127.0.0.1:8443":
```
frontend tls-in
    mode tcp
    bind :443
    default_backend alps

backend alps
    mode tcp
    server alps1 127.0.0.1:8443
```
Through a TCP passthrough, ALPS sees the passthrough's address on every connection. The access log then shows one address for all clients, and all users share one address's sign-in limits. With the default limits, that means 5 sign-in attempts a minute for the whole site, and 20 refused attempts in an hour, with no success in between, lock everyone out (see docs/RATELIMIT.md). trusted_proxies is meant for proxies that write the X-Forwarded-* headers themselves. A passthrough cannot write them, so do not list it there.
A proxy that terminates TLS on port 443 takes TLS-ALPN-01 away from ALPS and needs a certificate of its own anyway. In that setup, let the proxy hold the certificate and set [tls] enabled = false (Setup-Behind-a-Reverse-Proxy).

Related pages

Configuration-Overview: how the configuration file is read, and every section.
Config-Server: addr, trusted_proxies, the HTTP timeouts and debug.
Config-Cluster and Setup-Cluster: several nodes sharing Let's Encrypt certificates.
Config-Logging-and-Cache: where the log lines go.
Setup-Lets-Encrypt-Standalone: a single node with Let's Encrypt, step by step.
Setup-Behind-a-Reverse-Proxy: TLS in a proxy instead.
Running-in-Production: the checklist before going live.
docs/TLS.md, docs/CONFIGURATION.md, docs/OPERATIONS.md and docs/CLUSTER.md.

These pages describe ALPS as it is on the main branch; the reference documents live in the repository's docs/ directory.

Home

Architecture

Tests

Configuration

Common Setups

Config TLS

Choosing where TLS ends

[tls]

Certificate files (provider = "file")

Renewing the files

Errors that stop startup

Let's Encrypt (provider = "letsencrypt")

[tls.letsencrypt]

A single node

Challenges and ports

When certificates are ordered

Names and SNI

What ALPS stores

Checking that it works

Errors that stop startup

Staging and other directories

Sharing certificates through S3

[tls.letsencrypt.s3]

What happens

ACME through another server on port 80

Related pages

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

`[tls]`

Certificate files (`provider = "file"`)

Let's Encrypt (`provider = "letsencrypt"`)

`[tls.letsencrypt]`

`[tls.letsencrypt.s3]`