diff --git a/omni.yaml b/omni.yaml index 513e5238..bb8f59e6 100644 --- a/omni.yaml +++ b/omni.yaml @@ -24,6 +24,7 @@ navigation: - "omni-deployment-options.mdx" - "omni-on-prem-hardware-requirements.mdx" - "run-omni-on-prem.mdx" + - "run-an-etcd-cluster.mdx" - "run-omni-on-k8s.mdx" - "run-omni-airgapped.mdx" - "run-image-factory-on-prem.mdx" diff --git a/public/docs.json b/public/docs.json index 249b2f30..59a826dd 100644 --- a/public/docs.json +++ b/public/docs.json @@ -2608,6 +2608,7 @@ "omni/self-hosted/omni-deployment-options", "omni/self-hosted/omni-on-prem-hardware-requirements", "omni/self-hosted/run-omni-on-prem", + "omni/self-hosted/run-an-etcd-cluster", "omni/self-hosted/run-omni-on-k8s", "omni/self-hosted/run-omni-airgapped", "omni/self-hosted/run-image-factory-on-prem", diff --git a/public/omni/self-hosted/run-an-etcd-cluster.mdx b/public/omni/self-hosted/run-an-etcd-cluster.mdx new file mode 100644 index 00000000..dd727405 --- /dev/null +++ b/public/omni/self-hosted/run-an-etcd-cluster.mdx @@ -0,0 +1,420 @@ +--- +title: Run an etcd Cluster On-Prem +description: Deploy a 3-node etcd cluster secured with mTLS, for use as Omni's external datastore +--- + +This guide walks through deploying a 3-node etcd cluster secured with mutual TLS (mTLS). This cluster can be used as an external datastore for Omni on-prem, replacing Omni's embedded etcd for highly available deployments. + +Once complete, you will have: + +- A 3-node etcd cluster with mTLS enabled for both client and peer communication +- An etcd endpoint ready to connect to Omni + +A 3-node cluster requires a majority quorum to function. It can tolerate the loss of one node. For a production environment, run each node on a separate physical or virtual machine. + +## When to use external etcd + +Omni ships with embedded etcd and works well for most single-instance deployments. Use external etcd when you need: + +- **High availability** — survive the loss of an Omni host without losing datastore state +- **Operational separation** — back up, snapshot, and manage the datastore independently of Omni +- **Shared datastore** — run multiple Omni instances against the same etcd cluster + +See [Run Omni Options](./omni-deployment-options) for more context on when each mode is appropriate. + +## Prerequisites + +You will need **3 Linux hosts**, each with: + +- At least **2 vCPUs and 2 GB RAM** +- [Docker installed](https://docs.docker.com/engine/install/) +- The following ports open **between all 3 nodes**: + +| Port | Protocol | Purpose | +|------|----------|---------| +| 2379 | TCP | etcd client API (restrict to etcd node IPs and Omni host IP only) | +| 2380 | TCP | etcd peer communication (restrict to etcd node IPs only) | + +## Step 1: Install cfssl (Node 1 only) + +`cfssl` generates the CA and all node certificates. All certificates are created on Node 1 and then distributed to the other nodes in Step 4, so you only need cfssl on Node 1. + +```bash +CFSSL_VERSION=$(curl -sI https://github.com/cloudflare/cfssl/releases/latest \ + | grep -i location | awk -F '/' '{print $NF}' | tr -d '\r') + +curl -L -o cfssl \ + https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssl_${CFSSL_VERSION#v}_linux_amd64 +curl -L -o cfssljson \ + https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssljson_${CFSSL_VERSION#v}_linux_amd64 + +chmod +x cfssl cfssljson +sudo mv cfssl cfssljson /usr/local/bin/ +``` + +## Step 2: Set environment variables (Node 1 only) + +These variables define the IP addresses of all three etcd nodes and the Omni host. They are used when generating certificates and when starting the etcd cluster. Replace the values below with your actual IPs. + +```bash +export ETCD1_IP= +export ETCD2_IP= +export ETCD3_IP= +export OMNI_IP= +``` + +## Step 3: Generate certificates (Node 1 only) + +Each etcd node needs its own server certificate with its IP embedded as a Subject Alternative Name (SAN). This allows etcd to verify the identity of peer nodes and clients over mTLS. All certificates are signed by a shared root CA so every node in the cluster trusts the others. + +Omni also gets a dedicated client certificate, which it uses to authenticate to the cluster. + +### 3.1: Create the root CA + +This generates the root CA that all etcd nodes and Omni will use as their trust anchor. It produces `ca-key.pem` (private key), `ca.pem` (public cert), and `ca.csr` (signing request). + +```bash +cat < ca-csr.json +{ + "CN": "etcd Root CA", + "key": { "algo": "rsa", "size": 4096 }, + "names": [{ "C": "US", "O": "Internal Infrastructure", "OU": "Security" }] +} +EOF + +cfssl gencert -initca ca-csr.json | cfssljson -bare ca +``` + +### 3.2: Create the signing configuration + +The `server` profile is used for etcd node certificates and includes both `server auth` and `client auth` usages, since etcd nodes authenticate to each other as both clients and servers during peer communication. The `client` profile is used for the Omni client certificate. + +```bash +cat < ca-config.json +{ + "signing": { + "default": { "expiry": "8760h" }, + "profiles": { + "server": { + "usages": ["signing", "key encipherment", "server auth", "client auth"], + "expiry": "8760h" + }, + "client": { + "usages": ["signing", "key encipherment", "client auth"], + "expiry": "8760h" + } + } + } +} +EOF +``` + +### 3.3: Generate a server certificate for each node + +Each node gets its own certificate with its IP address embedded as a SAN. etcd uses these IPs to verify the identity of incoming peer and client connections. + +Run all three of the following commands on **Node 1**. + +The labels below indicate which node each certificate is for, **not** where the command runs. + +**Certificate for etcd-node-1:** + +```bash +cat < etcd1-csr.json +{ + "CN": "etcd-node-1", + "hosts": ["${ETCD1_IP}", "127.0.0.1"], + "key": { "algo": "rsa", "size": 4096 } +} +EOF + +cfssl gencert \ + -ca=ca.pem \ + -ca-key=ca-key.pem \ + -config=ca-config.json \ + -profile=server etcd1-csr.json | cfssljson -bare etcd1 +``` + +**Certificate for etcd-node-2:** + +```bash +cat < etcd2-csr.json +{ + "CN": "etcd-node-2", + "hosts": ["${ETCD2_IP}", "127.0.0.1"], + "key": { "algo": "rsa", "size": 4096 } +} +EOF + +cfssl gencert \ + -ca=ca.pem \ + -ca-key=ca-key.pem \ + -config=ca-config.json \ + -profile=server etcd2-csr.json | cfssljson -bare etcd2 +``` + +**Certificate for etcd-node-3:** + +```bash +cat < etcd3-csr.json +{ + "CN": "etcd-node-3", + "hosts": ["${ETCD3_IP}", "127.0.0.1"], + "key": { "algo": "rsa", "size": 4096 } +} +EOF + +cfssl gencert \ + -ca=ca.pem \ + -ca-key=ca-key.pem \ + -config=ca-config.json \ + -profile=server etcd3-csr.json | cfssljson -bare etcd3 +``` + +### 3.4: Generate the Omni client certificate + +This certificate is mounted into the Omni container and used to authenticate to etcd over mTLS. The `OMNI_IP` is embedded as a SAN so etcd can verify that connections from Omni are coming from the expected host. + +```bash +cat < client-csr.json +{ + "CN": "omni-client", + "hosts": ["${OMNI_IP}"], + "key": { "algo": "rsa", "size": 4096 }, + "names": [{ "C": "US", "O": "Omni" }] +} +EOF + +cfssl gencert \ + -ca=ca.pem \ + -ca-key=ca-key.pem \ + -config=ca-config.json \ + -profile=client client-csr.json | cfssljson -bare client + +chmod 644 client*.pem ca.pem +``` + +## Step 4: Distribute certificates + +In **Step 3**, all certificates were generated on Node 1. In this section, you’ll distribute those certificates to the other etcd and Omni nodes. + +Each node needs three items: its own certificate and private key (to prove its identity), and the shared CA certificate (to verify other nodes). The Omni host needs the client certificate and the CA certificate to authenticate with the cluster. + +The commands below use SSH agent forwarding, so your private key is never copied to any server. + +**On your local machine**, add your SSH key to the agent, then reconnect to Node 1 with forwarding enabled: + +```bash +ssh-add ~/path/to/your-ssh-key +ssh -A @ +``` + +Once connected to **Node 1**, run the following commands to distribute the certificates. If you started a new session, re-export the variables first: + +```bash +export ETCD2_IP= +export ETCD3_IP= +export OMNI_IP= +``` + +Create the destination directory on the Omni host: + +```bash +ssh @${OMNI_IP} "mkdir -p ~/etcd-certs" +``` + +Distribute the certificates to the other nodes: + +```bash +scp ca.pem etcd2.pem etcd2-key.pem @${ETCD2_IP}:~/ +scp ca.pem etcd3.pem etcd3-key.pem @${ETCD3_IP}:~/ +scp ca.pem client.pem client-key.pem @${OMNI_IP}:~/etcd-certs/ +``` + +Replace `` with your SSH username. Common values are `ubuntu` on Ubuntu, `ec2-user` on Amazon Linux and RHEL, and `root` on some bare metal setups. + +Keep `ca-key.pem` on Node 1 only. It is not needed on any other host and should not be distributed. + +## Step 5: Start etcd (all nodes) + +Each node starts etcd with its own certificate and the shared `--initial-cluster` flag that tells etcd the addresses of all three members. All three nodes must be started before the cluster can form quorum and become healthy. + +On each node, set the shared cluster variables: + +```bash +export ETCD1_IP= +export ETCD2_IP= +export ETCD3_IP= +``` + +Create the data directory where etcd will persist its state: + +```bash +mkdir -p $HOME/etcd-data +sudo chmod 700 $HOME/etcd-data +``` + +Then run the appropriate command for each node: + + + + +```bash +docker run -d \ + --name etcd \ + --restart=unless-stopped \ + -p 2379:2379 \ + -p 2380:2380 \ + -v $HOME/etcd-data:/etcd-data:Z \ + -v $HOME/etcd1.pem:/etcd/server.crt:ro \ + -v $HOME/etcd1-key.pem:/etcd/server.key:ro \ + -v $HOME/ca.pem:/etcd/ca.crt:ro \ + gcr.io/etcd-development/etcd:v3.5.17 \ + etcd \ + --name=etcd-node-1 \ + --data-dir=/etcd-data \ + --listen-client-urls=https://0.0.0.0:2379 \ + --advertise-client-urls=https://${ETCD1_IP}:2379 \ + --listen-peer-urls=https://0.0.0.0:2380 \ + --initial-advertise-peer-urls=https://${ETCD1_IP}:2380 \ + --initial-cluster="etcd-node-1=https://${ETCD1_IP}:2380,etcd-node-2=https://${ETCD2_IP}:2380,etcd-node-3=https://${ETCD3_IP}:2380" \ + --initial-cluster-state=new \ + --cert-file=/etcd/server.crt \ + --key-file=/etcd/server.key \ + --trusted-ca-file=/etcd/ca.crt \ + --client-cert-auth=true \ + --peer-cert-file=/etcd/server.crt \ + --peer-key-file=/etcd/server.key \ + --peer-trusted-ca-file=/etcd/ca.crt \ + --peer-client-cert-auth=true +``` + + + + +```bash +docker run -d \ + --name etcd \ + --restart=unless-stopped \ + -p 2379:2379 \ + -p 2380:2380 \ + -v $HOME/etcd-data:/etcd-data:Z \ + -v $HOME/etcd2.pem:/etcd/server.crt:ro \ + -v $HOME/etcd2-key.pem:/etcd/server.key:ro \ + -v $HOME/ca.pem:/etcd/ca.crt:ro \ + gcr.io/etcd-development/etcd:v3.5.17 \ + etcd \ + --name=etcd-node-2 \ + --data-dir=/etcd-data \ + --listen-client-urls=https://0.0.0.0:2379 \ + --advertise-client-urls=https://${ETCD2_IP}:2379 \ + --listen-peer-urls=https://0.0.0.0:2380 \ + --initial-advertise-peer-urls=https://${ETCD2_IP}:2380 \ + --initial-cluster="etcd-node-1=https://${ETCD1_IP}:2380,etcd-node-2=https://${ETCD2_IP}:2380,etcd-node-3=https://${ETCD3_IP}:2380" \ + --initial-cluster-state=new \ + --cert-file=/etcd/server.crt \ + --key-file=/etcd/server.key \ + --trusted-ca-file=/etcd/ca.crt \ + --client-cert-auth=true \ + --peer-cert-file=/etcd/server.crt \ + --peer-key-file=/etcd/server.key \ + --peer-trusted-ca-file=/etcd/ca.crt \ + --peer-client-cert-auth=true +``` + + + + +```bash +docker run -d \ + --name etcd \ + --restart=unless-stopped \ + -p 2379:2379 \ + -p 2380:2380 \ + -v $HOME/etcd-data:/etcd-data:Z \ + -v $HOME/etcd3.pem:/etcd/server.crt:ro \ + -v $HOME/etcd3-key.pem:/etcd/server.key:ro \ + -v $HOME/ca.pem:/etcd/ca.crt:ro \ + gcr.io/etcd-development/etcd:v3.5.17 \ + etcd \ + --name=etcd-node-3 \ + --data-dir=/etcd-data \ + --listen-client-urls=https://0.0.0.0:2379 \ + --advertise-client-urls=https://${ETCD3_IP}:2379 \ + --listen-peer-urls=https://0.0.0.0:2380 \ + --initial-advertise-peer-urls=https://${ETCD3_IP}:2380 \ + --initial-cluster="etcd-node-1=https://${ETCD1_IP}:2380,etcd-node-2=https://${ETCD2_IP}:2380,etcd-node-3=https://${ETCD3_IP}:2380" \ + --initial-cluster-state=new \ + --cert-file=/etcd/server.crt \ + --key-file=/etcd/server.key \ + --trusted-ca-file=/etcd/ca.crt \ + --client-cert-auth=true \ + --peer-cert-file=/etcd/server.crt \ + --peer-key-file=/etcd/server.key \ + --peer-trusted-ca-file=/etcd/ca.crt \ + --peer-client-cert-auth=true +``` + + + + +The `:Z` volume flag on all mounts ensures compatibility with SELinux. It is safe to use on non-SELinux hosts — Docker silently ignores it. + +## Step 6: Verify the cluster + +Once all three nodes are running, confirm they have formed a cluster. Run the following from Node 1. A healthy cluster will list all three members with `started` status and report each endpoint as healthy. + +Check that all three members have joined: + +```bash +docker exec etcd etcdctl \ + --endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379 \ + --cacert=/etcd/ca.crt \ + --cert=/etcd/server.crt \ + --key=/etcd/server.key \ + member list +``` + +**Expected output:** + +``` +, started, etcd-node-1, https://10.0.0.1:2380, https://10.0.0.1:2379, false +, started, etcd-node-2, https://10.0.0.2:2380, https://10.0.0.2:2379, false +, started, etcd-node-3, https://10.0.0.3:2380, https://10.0.0.3:2379, false +``` + +Check that all endpoints are healthy and accepting writes: + +```bash +docker exec etcd etcdctl \ + --endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379 \ + --cacert=/etcd/ca.crt \ + --cert=/etcd/server.crt \ + --key=/etcd/server.key \ + endpoint health +``` + +**Expected output:** + +``` +https://10.0.0.1:2379 is healthy: successfully committed proposal: took = ... +https://10.0.0.2:2379 is healthy: successfully committed proposal: took = ... +https://10.0.0.3:2379 is healthy: successfully committed proposal: took = ... +``` + +## Step 7: Connect Omni to the etcd cluster + +Once the cluster is healthy, connect Omni to it by passing the following flags when starting Omni. Refer to [Run Omni On-Prem](./run-omni-on-prem) and use the **External etcd** tab in Step 7. + +The `--etcd-embedded=false` flag disables Omni's internal etcd instance. The `--etcd-endpoints` flag points Omni at all three cluster members so it can fail over automatically if one node goes down. + +```bash +--etcd-embedded=false \ +--etcd-endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379 +``` + +Omni will use the client certificate you copied to `~/etcd-certs/` on the Omni host to authenticate to the cluster. + +## Backups + +Because etcd is external and not managed by Omni, you are responsible for backing it up independently. See [Back Up Omni Database](./back-up-omni-db) for instructions on taking etcd snapshots. diff --git a/public/omni/self-hosted/run-omni-on-prem.mdx b/public/omni/self-hosted/run-omni-on-prem.mdx index ab2a1a1a..ab9c0fa1 100644 --- a/public/omni/self-hosted/run-omni-on-prem.mdx +++ b/public/omni/self-hosted/run-omni-on-prem.mdx @@ -1,275 +1,512 @@ --- title: Run Omni On-Prem -description: Run Omni on your infrastructure +description: Deploy a self-hosted Omni instance on any internet-connected Linux host --- -import { omni_release } from '/snippets/custom-variables.mdx'; +This guide walks you through deploying Omni on a Linux host with internet access. By the end, you will have a fully functional Omni instance running with TLS, an OIDC identity provider, and either embedded or external etcd. -This guide shows you how to run Omni on-prem. This guide assumes that Omni will be deployed on an Ubuntu machine. Small differences should be expected when using a different OS. +Omni is licensed under the [Business Source License](https://github.com/siderolabs/omni/blob/main/LICENSE) and requires a support contract for production use. Contact [Sidero sales](mailto:sales@siderolabs.com) for a production license. -For SAML integration sections, this guide assumes Azure AD will be the provider for SAML. +## Prerequisites -Omni is available via a [Business Source License](https://github.com/siderolabs/omni/blob/main/LICENSE) which allows free installations in non-production environments. If you would like to deploy Omni for production use please contact [Sidero sales](mailto:sales@siderolabs.com?subject=Omni%20license%20inquiry\&body=Hello,%20I%20would%20like%20to%20purchase%20an%20on-prem%20license%20for%20Omni.). If you would like to subscribe to the hosted version of Omni please see the [SaaS pricing](https://www.siderolabs.com/pricing/); +You will need: -### Prerequisites +- A Linux host with at least **2 vCPUs and 4 GB RAM** +- A domain name or IP address for your Omni instance +- **Docker** installed on the host +- The following ports open and accessible: -There are several prerequisites for deploying Omni on-prem. We will assume you have an Ubuntu machine available. Any distribution with Docker should work. +| Port | Protocol | Purpose | +|------|----------|---------| +| 443 | TCP | Omni UI and API | +| 8090 | TCP | SideroLink API | +| 8091 | TCP | Event sink | +| 8100 | TCP | Kubernetes proxy | +| 5556 | TCP | Dex OIDC | +| 50180 | UDP | WireGuard | -#### Install Docker +## Step 1: Install Docker -Install Docker according to the Ubuntu installation guide [here](https://docs.docker.com/engine/install/ubuntu/). You will also need the docker compose plugin package if you want to use the example docker compose template. +Omni and its supporting services run as Docker containers. Follow the [official Docker installation guide](https://docs.docker.com/engine/install/) for your Linux distribution. +After installing, add your user to the `docker` group so you can run Docker without `sudo`: + +```bash +sudo usermod -aG docker $USER +newgrp docker +``` + +## Step 2: Install cfssl + +`cfssl` generates and signs the TLS certificates used by Omni and Dex. + +```bash +CFSSL_VERSION=$(curl -sI https://github.com/cloudflare/cfssl/releases/latest \ + | grep -i location | awk -F '/' '{print $NF}' | tr -d '\r') + +curl -L -o cfssl \ + https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssl_${CFSSL_VERSION#v}_linux_amd64 +curl -L -o cfssljson \ + https://github.com/cloudflare/cfssl/releases/download/${CFSSL_VERSION}/cfssljson_${CFSSL_VERSION#v}_linux_amd64 + +chmod +x cfssl cfssljson +sudo mv cfssl cfssljson /usr/local/bin/ +``` + +## Step 3: Set environment variables + +Set variables that define your host's network addresses and the internal hostnames for Omni and Dex. They are referenced throughout the guide, so set them before proceeding. + +```bash +export HOST_PUBLIC_IP=$(curl -s https://ifconfig.me) +export HOST_PRIVATE_IP=$(hostname -I | awk '{print $1}') +export OMNI_ENDPOINT=omni.internal +export AUTH_ENDPOINT=auth.internal +export OMNI_USER_EMAIL="admin@omni.internal" + +echo "Public IP: $HOST_PUBLIC_IP" +echo "Private IP: $HOST_PRIVATE_IP" +``` + +If your host does not have a public IP (e.g. a local VM), set `HOST_PUBLIC_IP` to the IP your local machine uses to reach the host. + +### 3.1: Add internal hostnames to `/etc/hosts` + +Omni and Dex use `.internal` hostnames that are not registered in public DNS. This maps them to localhost so the host can resolve them. + +```bash +echo "127.0.0.1 ${OMNI_ENDPOINT} ${AUTH_ENDPOINT}" | sudo tee -a /etc/hosts +``` + +## Step 4: Generate TLS certificates + +Omni and Dex communicate over TLS. In this step, you create a root CA and use it to sign a certificate covering both service endpoints. The CA certificate is also distributed to client machines in **Step 9** so their browsers trust the Omni UI. + +If you already have a trusted internal CA, use it to sign the certificate in **Step 4.3** and skip **Steps 4.1 and 4.2**. + +### 4.1: Create the root CA + +Run this command to generate a self-signed root CA, the trust anchor for all certificates in this setup. + +It produces `ca-key.pem` (private key), `ca.pem` (public cert), and `ca.csr` (signing request). + +```bash +cat < ca-csr.json +{ + "CN": "Internal Root CA", + "key": { "algo": "rsa", "size": 4096 }, + "names": [{ "C": "US", "O": "Internal Infrastructure", "OU": "Security" }] +} +EOF + +cfssl gencert -initca ca-csr.json | cfssljson -bare ca +``` + +Add the CA to the host's system trust store so services running on this machine trust certificates signed by it: + +```bash +sudo cp ca.pem /usr/local/share/ca-certificates/ca.crt +sudo update-ca-certificates +``` + +### 4.2: Create the signing configuration + +Create a signing configuration that tells `cfssl` how to sign certificates. The `web-server` profile is used for Omni and Dex. The `client` profile is used later for external etcd authentication in **Step 7**. + +```bash +cat < ca-config.json +{ + "signing": { + "default": { "expiry": "8760h" }, + "profiles": { + "web-server": { + "usages": ["signing", "key encipherment", "server auth"], + "expiry": "8760h" + }, + "client": { + "usages": ["signing", "key encipherment", "client auth"], + "expiry": "8760h" + } + } + } +} +EOF +``` + +### 4.3: Generate the server certificate + +Generate a single wildcard certificate covering both the Omni and Dex endpoints. Both IPs are included so TLS connections work regardless of whether clients connect by hostname or IP. + +```bash +cat < wildcard-csr.json +{ + "CN": "Internal Wildcard", + "hosts": [ + "${OMNI_ENDPOINT}", + "${AUTH_ENDPOINT}", + "127.0.0.1", + "${HOST_PUBLIC_IP}", + "${HOST_PRIVATE_IP}" + ], + "key": { "algo": "rsa", "size": 4096 } +} +EOF +``` + +Sign the certificate using the CA and configuration from the previous steps: + +```bash +cfssl gencert \ + -ca=ca.pem \ + -ca-key=ca-key.pem \ + -config=ca-config.json \ + -profile=web-server wildcard-csr.json | cfssljson -bare server +``` + +Bundle the server certificate with the CA so clients can verify the full chain: + +```bash +cat server.pem ca.pem > server-chain.pem ``` -curl -L https://get.docker.io | sh + +This produces `server-key.pem` (private key), `server.pem` (public cert), and `server-chain.pem` (cert bundled with CA). Make them readable by all service users by running: + +```bash +chmod 644 server*.pem ``` -#### Generate certificates +## Step 5: Generate the etcd encryption key + +Omni encrypts all data written to etcd at rest using a GPG key. This generates the key and exports it to `omni.asc`, which Omni reads on startup. -On-prem Omni will require valid SSL certificates. This means that self-signed certs _will not_ work. Generating certificates is left as an exercise to the user, but here is a rough example that was tested using [DigitalOcean's DNS integration](https://certbot-dns-digitalocean.readthedocs.io/en/stable/) with certbot to generate certificates. The process should be very similar for other providers like Route53. +Do not set a passphrase on this key. Omni reads the key file on startup and cannot prompt for a passphrase interactively. ```bash -# Install certbot -$ sudo snap install --classic certbot +gpg --batch --passphrase '' \ + --quick-generate-key \ + "Omni (Used for etcd data encryption) omni@internal.local" \ + rsa4096 cert never + +FINGERPRINT=$(gpg --with-colons --list-keys "omni@internal.local" \ + | awk -F: '$1 == "fpr" {print $10; exit}') + +gpg --batch --passphrase '' \ + --quick-add-key ${FINGERPRINT} rsa4096 encr never + +gpg --export-secret-key --armor omni@internal.local > omni.asc +``` + +## Step 6: Set up an identity provider + +Omni requires an OIDC or SAML identity provider for user authentication. This guide uses [Dex](https://dexidp.io/), configured with a static user. Dex can also act as a proxy to upstream providers such as LDAP, SAML, GitHub, and Okta, see the [Dex connector documentation](https://dexidp.io/docs/connectors/) for details. + +If you already have an existing SAML or OIDC provider and want to connect it directly to Omni, skip this step and see [Authentication and Authorization](../security-and-authentication/authentication-and-authorization). + +### 6.1: Create a password hash for the admin user + +You will be prompted to enter a password. This becomes your Omni login password. The hash is stored in the Dex configuration and never transmitted in plain text. -# Allow for root access -$ sudo snap set certbot trust-plugin-with-root=ok +```bash +export OMNI_USER_PASSWORD=$(docker run --rm httpd:2.4-alpine \ + htpasswd -BnC 15 admin | cut -d: -f2) +``` -# Install DNS provider -$ snap install certbot-dns- +### 6.2: Write the Dex configuration -# Create creds file with API tokens -$ echo ' creds.ini +Create a `dex.yaml` configuration file, which configures Dex to serve OIDC over TLS on port 5556, registers Omni as an OAuth2 client, and creates a single static login user. -# Create certs for desired domain -$ certbot certonly --dns- -d +```bash +cat < dex.yaml +issuer: https://${AUTH_ENDPOINT}:5556 + +storage: + type: memory + +web: + https: 0.0.0.0:5556 + tlsCert: /etc/dex/tls/server-chain.pem + tlsKey: /etc/dex/tls/server-key.pem + +enablePasswordDB: true + +staticClients: + - name: Omni + id: omni + secret: omni-dex-secret + redirectURIs: + - https://${OMNI_ENDPOINT}/oidc/consume + +staticPasswords: + - email: "${OMNI_USER_EMAIL}" + username: "admin" + preferredUsername: "admin" + hash: "${OMNI_USER_PASSWORD}" +EOF ``` -### Configure authentication +### 6.3: Run Dex -#### Auth0 +Start Dex as a container, mounting the configuration file and TLS certificates. The `:Z` flag on volume mounts ensures compatibility with SELinux if it is enabled on your host. -Create an [Auth0 account](https://auth0.com/signup). +```bash +docker run -d \ + --name dex \ + --restart=unless-stopped \ + -p 5556:5556 \ + -v $(pwd)/dex.yaml:/etc/dex/dex.yaml:ro,Z \ + -v $(pwd)/server-key.pem:/etc/dex/tls/server-key.pem:ro,Z \ + -v $(pwd)/server-chain.pem:/etc/dex/tls/server-chain.pem:ro,Z \ + ghcr.io/dexidp/dex:v2.41.1 \ + dex serve /etc/dex/dex.yaml +``` -On the account level, configure "Authentication - Social" to allow GitHub and Google login. +## Step 7: Run Omni -Create an Auth0 application of the type "single page web application". +In this step you will start the Omni container and connect it to the services you set up in the previous steps: the TLS certificates from **Step 4**, the encryption key from **Step 5**, and the identity provider from **Step 6**. -Configure the Auth0 application with the following: +### 7.1: Get the latest Omni version -* Allowed callback URLs: `https://` -* Allowed web origins: `https://` -* Allowed logout URLs: `https://` +Fetch the latest Omni release tag from GitHub and store it in `OMNI_VERSION`: -Disable username/password auth on "Authentication - Database - Applications" tab. +```bash +export OMNI_VERSION=$(curl -sI https://github.com/siderolabs/omni/releases/latest \ + | grep -i location | awk -F '/' '{print $NF}' | tr -d '\r') -Enable GitHub and Google login on the "Authentication - Social" tab. +echo "Using Omni version: $OMNI_VERSION" +``` -Enable email access in the GitHub settings. +### 7.2: Create the SQLite storage directory -Take note of the following information from the Auth0 application: +Omni uses SQLite alongside etcd for certain internal state. -* Domain -* Client ID +This command creates the directory on the host that will be mounted into the container: -#### SAML identity providers +```bash +mkdir -p $HOME/sqlite +``` -Other identity providers also work with Omni. Configuring these should be similar to Auth0. +### 7.3: Start Omni -* [EntraID/Azure AD](../security-and-authentication/using-saml-with-omni/how-to-configure-entraid-for-omni) -* [Keycloak](./configure-keycloak-for-omni) -* [Okta](../security-and-authentication/using-saml-with-omni/configure-okta-for-omni) -* [Workspace ONE Access](../security-and-authentication/using-saml-with-omni/configure-workspace-one-access-for-omni) -* [Unifi Identity Enterprise](../security-and-authentication/using-saml-with-omni/configure-unifi-identity-enterprise-for-omni) +Choose the etcd mode that matches your setup. -### Create etcd encryption key +Embedded etcd is suitable for a single Omni instance. External etcd is required for high availability or if you need to manage the datastore independently. -Generate a GPG key: + + ```bash -gpg --quick-generate-key "Omni (Used for etcd data encryption) how-to-guide@siderolabs.com" rsa4096 cert never +docker run -d \ + --name omni \ + --net=host \ + --cap-add=NET_ADMIN \ + --device /dev/net/tun:/dev/net/tun \ + --restart=unless-stopped \ + -v $(pwd)/ca.pem:/etc/ssl/certs/ca-certificates.crt:ro,Z \ + -v $(pwd)/server-key.pem:/server-key.pem:ro,Z \ + -v $(pwd)/server-chain.pem:/server-chain.pem:ro,Z \ + -v $(pwd)/omni.asc:/omni.asc:ro,Z \ + -v $HOME/sqlite:/_out/sqlite:rw,Z \ + ghcr.io/siderolabs/omni:${OMNI_VERSION} \ + --name=omni \ + --cert=/server-chain.pem \ + --key=/server-key.pem \ + --machine-api-cert=/server-chain.pem \ + --machine-api-key=/server-key.pem \ + --machine-api-bind-addr=0.0.0.0:8090 \ + --private-key-source=file:///omni.asc \ + --event-sink-port=8091 \ + --bind-addr=0.0.0.0:443 \ + --k8s-proxy-bind-addr=0.0.0.0:8100 \ + --advertised-api-url=https://${OMNI_ENDPOINT}/ \ + --siderolink-api-advertised-url=https://${OMNI_ENDPOINT}:8090/ \ + --siderolink-wireguard-advertised-addr=${HOST_PUBLIC_IP}:50180 \ + --advertised-kubernetes-proxy-url=https://${OMNI_ENDPOINT}:8100/ \ + --auth-auth0-enabled=false \ + --auth-oidc-enabled=true \ + --auth-oidc-provider-url=https://${AUTH_ENDPOINT}:5556 \ + --auth-oidc-client-id=omni \ + --auth-oidc-client-secret=omni-dex-secret \ + --auth-oidc-scopes=openid \ + --auth-oidc-scopes=profile \ + --auth-oidc-scopes=email \ + --sqlite-storage-path=/_out/sqlite/omni.db \ + --initial-users=${OMNI_USER_EMAIL} ``` + + + + Use this if you have a completed external etcd cluster. See [Run an etcd Cluster On-Prem](./run-an-etcd-cluster) to set one up first. + + Set the private IPs of your etcd nodes. + + ```bash + export ETCD1_IP= + export ETCD2_IP= + export ETCD3_IP= + ``` + + Start Omni with the external etcd flags. The `--etcd-embedded=false` flag disables the internal etcd instance. The `--etcd-endpoints` flag lists all three nodes so Omni can fail over automatically if one goes down: + + ```bash + docker run -d \ + --name omni \ + --net=host \ + --cap-add=NET_ADMIN \ + --device /dev/net/tun:/dev/net/tun \ + --restart=unless-stopped \ + -v $(pwd)/ca.pem:/etc/ssl/certs/ca-certificates.crt:ro,Z \ + -v $(pwd)/server-key.pem:/server-key.pem:ro,Z \ + -v $(pwd)/server-chain.pem:/server-chain.pem:ro,Z \ + -v $(pwd)/omni.asc:/omni.asc:ro,Z \ + -v $HOME/etcd-certs/ca.pem:/etcd/ca.crt:ro,Z \ + -v $HOME/etcd-certs/client.pem:/etcd/client.crt:ro,Z \ + -v $HOME/etcd-certs/client-key.pem:/etcd/client.key:ro,Z \ + -v $HOME/sqlite:/_out/sqlite:rw,Z \ + ghcr.io/siderolabs/omni:${OMNI_VERSION} \ + --name=omni \ + --cert=/server-chain.pem \ + --key=/server-key.pem \ + --machine-api-cert=/server-chain.pem \ + --machine-api-key=/server-key.pem \ + --machine-api-bind-addr=0.0.0.0:8090 \ + --private-key-source=file:///omni.asc \ + --event-sink-port=8091 \ + --bind-addr=0.0.0.0:443 \ + --k8s-proxy-bind-addr=0.0.0.0:8100 \ + --advertised-api-url=https://${OMNI_ENDPOINT}/ \ + --siderolink-api-advertised-url=https://${OMNI_ENDPOINT}:8090/ \ + --siderolink-wireguard-advertised-addr=${HOST_PUBLIC_IP}:50180 \ + --advertised-kubernetes-proxy-url=https://${OMNI_ENDPOINT}:8100/ \ + --etcd-embedded=false \ + --etcd-endpoints=https://${ETCD1_IP}:2379,https://${ETCD2_IP}:2379,https://${ETCD3_IP}:2379 \ + --auth-auth0-enabled=false \ + --auth-oidc-enabled=true \ + --auth-oidc-provider-url=https://${AUTH_ENDPOINT}:5556 \ + --auth-oidc-client-id=omni \ + --auth-oidc-client-secret=omni-dex-secret \ + --auth-oidc-scopes=openid \ + --auth-oidc-scopes=profile \ + --auth-oidc-scopes=email \ + --sqlite-storage-path=/_out/sqlite/omni.db \ + --initial-users=${OMNI_USER_EMAIL} + ``` -Find the fingerprint of the generated key: + + + +## Step 8: Verify services are running + +Check that both Omni and Dex containers started successfully: ```bash -gpg --list-secret-keys +docker ps --format "table {{.Names}}\t{{.Status}}" ``` -Using the fingerprint, add an encryption subkey and export: +**Expected output:** + +``` +NAMES STATUS +omni Up X minutes +dex Up X minutes +``` + +Confirm Omni is serving on port 443: ```bash -gpg --quick-add-key rsa4096 encr never -gpg --export-secret-key --armor how-to-guide@siderolabs.com > omni.asc +curl -k https://127.0.0.1:443 ``` -**Note:** Do not add passphrases to keys during creation. +This should return the Omni HTML page. + +## Step 9: Access the Omni UI -### Deploy Omni +Omni is running on your host, but your local machine does not yet trust the CA certificate or know how to resolve the internal hostnames. The steps below set that up. -There are two easy ways to run Omni: docker-compose and a simple `docker run`. We recommend using docker-compose, but both are detailed in separate tabs below. +### 9.1: Copy the CA certificate to your local machine + +Copy `ca.pem` from the host to your local machine. You will install it as a trusted root in the next step. + - - #### Export variables + Replace `` with your instance's public IP and `~/path/to/your-key.pem` with your SSH key path. - You will need to specify some customizations for your installation. Export these variables with your information to use in the provided templates + ```bash + scp -i ~/path/to/your-key.pem ubuntu@:~/ca.pem ~/Downloads/omni-ca.pem + ``` + + - If you are running multiple Omni instances connected to a shared etcd backend you will need to set the `accountUUID` to a unique value. If you are not using shared backend storage then this option does not need to be unique. + Replace `` and `` with your GCP instance name and zone. - - {`export OMNI_VERSION=${omni_release}\nOMNI_ACCOUNT_UUID=$(uuidgen)\nOMNI_DOMAIN_NAME=omni.siderolabs.com\nOMNI_WG_IP=10.10.1.100\nOMNI_ADMIN_EMAIL=omni@siderolabs.com\nAUTH0_CLIENT_ID=xxxyyyzzz\nAUTH0_DOMAIN=dev-aaabbbccc.us.auth0.com - `} - + ```bash + gcloud compute scp :~/ca.pem ~/Downloads/omni-ca.pem --zone= + ``` + + - #### Download Assets + Replace `` with your host's IP. - To pull down the Omni deployment assets for Docker Compose, simply grab them with curl as follows. + ```bash + scp ubuntu@:~/ca.pem ~/Downloads/omni-ca.pem + ``` + + - ```bash - curl https://raw.githubusercontent.com/siderolabs/omni/${OMNI_VERSION}/deploy/env.template \ - | envsubst > omni.env +### 9.2: Trust the CA certificate - curl https://raw.githubusercontent.com/siderolabs/omni/${OMNI_VERSION}/deploy/compose.yaml -o compose.yaml - ``` +Add the CA to your local machine's system trust store. This tells your browser and OS to trust TLS certificates signed by this CA, including the Omni and Dex endpoints. - #### Verify settings + + + On Red Hat and Fedora based distros you can copy the `ca.pem` file into the `/etc/pki/ca-trust/source/anchors/` folder and then run the following command to generate the trusted root store: - Open the omni.env file to check that all of your variables have been set to your environment requirements. + ```bash + sudo cp ca.pem /etc/pki/ca-trust/source/anchors/ + sudo update-ca-trust + ``` + + + On Ubuntu and Debian based Linux distros you can copy the `ca.pem` file into the `/usr/local/share/ca-certificates/` directory and rename it to `ca.crt` + + ```bash + sudo cp ca.pem /usr/local/share/ca-certificates/ca.crt + sudo update-ca-certificates + ``` + + + For macOS you should open the *Keychain Access* application and drag the `ca.pem` file into the window to install it. + + + On Windows you should rename the certificate extension from `.pem` to `.crt`. You can then double click on the file and select *Install Certificate* -> *Local Machine*. You then need to select "Place all certificates in the following store" and select the *Trusted Root Certification Authorities*. + + If you're using Windows Subsystem for Linux (WSL) you should follow the Linux guide for installing the certificate. + + + +### 9.3: Add internal hostnames to your local hosts file - #### Run It! +Map the Omni and Dex hostnames to your host's public IP so your local machine can resolve them. Replace `` with your host's public IP. - With your environment file in hand, it's now time to run Omni. This can be done with a simple docker compose command: + + - ``` - docker compose --env-file omni.env up -d - ``` + ```bash + sudo sh -c 'echo " omni.internal auth.internal" >> /etc/hosts' + ``` + - - Deploying with a `docker run` is quite straight forward, with only some slight differences depending on the auth mechanism in use. - - #### Auth0 - - ```bash - docker run \ - --net=host \ - --cap-add=NET_ADMIN \ - --device /dev/net/tun \ - -v $PWD/etcd:/_out/etcd \ - -v :/tls.crt \ - -v :/tls.key \ - -v $PWD/omni.asc:/omni.asc \ - ghcr.io/siderolabs/omni: \ - --account-id=$(uuidgen) \ - --name=onprem-omni \ - --cert=/tls.crt \ - --key=/tls.key \ - --siderolink-api-cert=/tls.crt \ - --siderolink-api-key=/tls.key \ - --private-key-source=file:///omni.asc \ - --event-sink-port=8091 \ - --bind-addr=0.0.0.0:443 \ - --siderolink-api-bind-addr=0.0.0.0:8090 \ - --k8s-proxy-bind-addr=0.0.0.0:8100 \ - --advertised-api-url=https:/// \ - --siderolink-api-advertised-url=https://:8090/ \ - --siderolink-wireguard-advertised-addr=:50180 \ - --advertised-kubernetes-proxy-url=https://:8100/ \ - --auth-auth0-enabled=true \ - --auth-auth0-domain= \ - --auth-auth0-client-id= \ - --initial-users= - ``` - - **Note:** The `siderolink-wireguard-advertised-addr` **must** point to an IP, not the domain name. - - **Note:** you can omit the `--cert`, `--key`, `--siderolink-api-cert`, and `--siderolink-api-key` flags to run Omni insecurely. - - Configuration options are available in the help menu (`--help`). - - #### SAML - - ```bash - docker run \ - --net=host \ - --cap-add=NET_ADMIN \ - --device /dev/net/tun \ - -v $PWD/etcd:/_out/etcd \ - -v :/tls.crt \ - -v :/tls.key \ - -v $PWD/omni.asc:/omni.asc \ - ghcr.io/siderolabs/omni: \ - --account-id=$(uuidgen) \ - --name=onprem-omni \ - --cert=/tls.crt \ - --key=/tls.key \ - --siderolink-api-cert=/tls.crt \ - --siderolink-api-key=/tls.key \ - --private-key-source=file:///omni.asc \ - --event-sink-port=8091 \ - --bind-addr=0.0.0.0:443 \ - --siderolink-api-bind-addr=0.0.0.0:8090 \ - --k8s-proxy-bind-addr=0.0.0.0:8100 \ - --advertised-api-url=https:/// \ - --siderolink-api-advertised-url=https://:8090/ \ - --siderolink-wireguard-advertised-addr=:50180 \ - --advertised-kubernetes-proxy-url=https://:8100/ \ - --auth-saml-enabled=true \ - --auth-saml-url= - ``` - - **Note** - - In a default setup, the first user that logs in via SAML will be the “admin”. All subsequent users will receive a read-only role and may need to be granted additional access by the admin user from the “Users” tab in Omni. - - #### OIDC - - ```bash - docker run \ - --net=host \ - --cap-add=NET_ADMIN \ - --device /dev/net/tun \ - -v $PWD/etcd:/_out/etcd \ - -v :/tls.crt \ - -v :/tls.key \ - -v $PWD/omni.asc:/omni.asc \ - ghcr.io/siderolabs/omni: \ - --account-id=$(uuidgen) \ - --name=onprem-omni \ - --cert=/tls.crt \ - --key=/tls.key \ - --siderolink-api-cert=/tls.crt \ - --siderolink-api-key=/tls.key \ - --private-key-source=file:///omni.asc \ - --event-sink-port=8091 \ - --bind-addr=0.0.0.0:443 \ - --siderolink-api-bind-addr=0.0.0.0:8090 \ - --k8s-proxy-bind-addr=0.0.0.0:8100 \ - --advertised-api-url=https:/// \ - --siderolink-api-advertised-url=https://:8090/ \ - --siderolink-wireguard-advertised-addr=:50180 \ - --advertised-kubernetes-proxy-url=https://:8100/ \ - --auth-oidc-enabled \ - --auth-oidc-provider-url \ - --auth-oidc-client-id \ - --auth-oidc-client-secret \ - --auth-oidc-logout-url \ - --auth-oidc-scopes openid \ - --auth-oidc-scopes profile \ - --auth-oidc-scopes email - ``` - - **Note** - - The `siderolink-wireguard-advertised-addr` **must** point to an IP, not the domain name. - - **Note** - - Note that you can omit the `--cert`, `--key`, `--siderolink-api-cert`, and `--siderolink-api-key` flags to run Omni insecurely. - - Configuration options are available in the help menu (`--help`). + Add the following line to `C:\Windows\System32\drivers\etc\hosts`: + + ``` + omni.internal auth.internal + ``` + +### 9.5: Log into Omni + +Open `https://omni.internal` in your browser and sign in with the email and password you set in **Step 6.2**.