From 54150b108d90821f596f61a4faa52592d6217fb8 Mon Sep 17 00:00:00 2001 From: Caleb Doxsey Date: Wed, 17 Sep 2025 14:10:08 -0600 Subject: [PATCH 1/5] add clustered databroker documentation --- content/docs/reference/databroker.mdx | 137 ++++++++++++++++++++++++++ 1 file changed, 137 insertions(+) diff --git a/content/docs/reference/databroker.mdx b/content/docs/reference/databroker.mdx index 82a734352..5c4afced8 100644 --- a/content/docs/reference/databroker.mdx +++ b/content/docs/reference/databroker.mdx @@ -145,3 +145,140 @@ See Kubernetes [Storage reference](/docs/deploy/k8s/reference#storage) for more When using multiple hosts make sure to specify `target_session_attrs=read-write` so that the Databroker does not attempt to write to a read-only replica. ::: + +## Clustered Databroker + +As of v0.31, Pomerium supports an experimental clustered databroker. The clustered databroker consists of multiple databroker instances. One of those instances is the cluster leader and all the other instances are cluster followers. Databroker commands sent to a follower are forwarded to the leader where they are handled. In addition the clustered databroker supports automatic failure recovery via Raft leader election. + +Each databroker instance should have the same shared secret, but its own node id and its own storage backend (the instances should **not** share the same Postgres database). + +The following settings are supported by the clustered databroker: + +### Databroker Cluster Node ID + +The Databroker Cluster Node ID sets the databroker's node id. It should correspond to an entry in the Databroker Cluster Nodes option, and it should be unique for each instance of the databroker. + + + + +| **Config file keys** | **Environment variables** | **Type** | +| :--------------------------- | :--------------------------- | :------- | +| `databroker_cluster_node_id` | `DATABROKER_CLUSTER_NODE_ID` | `string` | + +```yaml +databroker_cluster_node_id: node-1 +``` + +```bash +DATABROKER_CLUSTER_NODE_ID=node-1 +``` + + + +`databroker_cluster_node_id` is a bootstrap configuration setting and is not configurable in the Console. + + + + +`databroker_cluster_node_id` is not supported in Kubernetes. + + + + +### Databroker Cluster Leader ID + +The Databroker Cluster Leader ID explicitly sets the leader to one of the databroker instances in the Databroker Cluster Nodes option. When this option is used, the Raft leader elector will not be used and a failure of the leader will not result in automatic failure recovery. + + + + +| **Config file keys** | **Environment variables** | **Type** | +| :----------------------------- | :----------------------------- | :------- | +| `databroker_cluster_leader_id` | `DATABROKER_CLUSTER_LEADER_ID` | `string` | + +```yaml +databroker_cluster_leader_id: node-2 +``` + +```bash +DATABROKER_CLUSTER_LEADER_ID=node-2 +``` + + + +`databroker_cluster_leader_id` is a bootstrap configuration setting and is not configurable in the Console. + + + + +`databroker_cluster_leader_id` is not supported in Kubernetes. + + + + +### Databroker Cluster Nodes + +The Databroker Cluster Nodes option defines the cluster topology of a clustered databroker. It consists of a list of node definitions, each of which has an `id`, `grpc_address` and `raft_address`. Each instance of the databroker that is part of the cluster should have the same cluster nodes definition. When using the Raft leader elector there should be an odd number of at least 3 nodes. + + + + +| **Config file keys** | **Environment variables** | **Type** | +| :------------------------- | :------------------------- | :------- | +| `databroker_cluster_nodes` | `DATABROKER_CLUSTER_NODES` | `string` | + +```yaml +databroker_cluster_nodes: + - id: node-1 + grpc_address: http://node-1:5443 + raft_address: node-1:5999 + - id: node-2 + grpc_address: http://node-2:5443 + raft_address: node-2:5999 + - id: node-3 + grpc_address: http://node-3:5443 + raft_address: node-2:5999 +``` + + + +`databroker_cluster_nodes` is a bootstrap configuration setting and is not configurable in the Console. + + + + +`databroker_cluster_nodes` is not supported in Kubernetes. + + + + +### Databroker Raft Bind Address + +To use the Raft leader elector, each node needs to set a raft bind address which will be used for TCP connections between each instance. This communication is automatically encrypted using TLS with certificates derived from the shared secret. The `raft_address` in the `databroker_cluster_nodes` option for this node should correspond to this address. + + + + +| **Config file keys** | **Environment variables** | **Type** | +| :----------------------------- | :----------------------------- | :------- | +| `databroker_raft_bind_address` | `DATABROKER_RAFT_BIND_ADDRESS` | `string` | + +```yaml +databroker_raft_bind_address: :5999 +``` + +```bash +DATABROKER_RAFT_BIND_ADDRESS=:5999 +``` + + + +`databroker_raft_bind_address` is a bootstrap configuration setting and is not configurable in the Console. + + + + +`databroker_raft_bind_address` is not supported in Kubernetes. + + + From c0a212753a8862cd4f4d40c0418fea962dd007bf Mon Sep 17 00:00:00 2001 From: Caleb Doxsey Date: Thu, 18 Sep 2025 12:51:08 -0600 Subject: [PATCH 2/5] fix missing closing tabs --- content/docs/reference/databroker.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/docs/reference/databroker.mdx b/content/docs/reference/databroker.mdx index 5c4afced8..87a2f973b 100644 --- a/content/docs/reference/databroker.mdx +++ b/content/docs/reference/databroker.mdx @@ -173,6 +173,7 @@ databroker_cluster_node_id: node-1 DATABROKER_CLUSTER_NODE_ID=node-1 ``` + `databroker_cluster_node_id` is a bootstrap configuration setting and is not configurable in the Console. @@ -204,6 +205,7 @@ databroker_cluster_leader_id: node-2 DATABROKER_CLUSTER_LEADER_ID=node-2 ``` + `databroker_cluster_leader_id` is a bootstrap configuration setting and is not configurable in the Console. @@ -240,6 +242,7 @@ databroker_cluster_nodes: raft_address: node-2:5999 ``` + `databroker_cluster_nodes` is a bootstrap configuration setting and is not configurable in the Console. @@ -271,6 +274,7 @@ databroker_raft_bind_address: :5999 DATABROKER_RAFT_BIND_ADDRESS=:5999 ``` + `databroker_raft_bind_address` is a bootstrap configuration setting and is not configurable in the Console. From 7911037add90b8b2300d2d8a57990e7281360605 Mon Sep 17 00:00:00 2001 From: Caleb Doxsey Date: Tue, 30 Sep 2025 09:08:37 -0600 Subject: [PATCH 3/5] add recommendation --- content/docs/reference/databroker.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/reference/databroker.mdx b/content/docs/reference/databroker.mdx index 87a2f973b..a5551b25b 100644 --- a/content/docs/reference/databroker.mdx +++ b/content/docs/reference/databroker.mdx @@ -150,7 +150,7 @@ When using multiple hosts make sure to specify `target_session_attrs=read-write` As of v0.31, Pomerium supports an experimental clustered databroker. The clustered databroker consists of multiple databroker instances. One of those instances is the cluster leader and all the other instances are cluster followers. Databroker commands sent to a follower are forwarded to the leader where they are handled. In addition the clustered databroker supports automatic failure recovery via Raft leader election. -Each databroker instance should have the same shared secret, but its own node id and its own storage backend (the instances should **not** share the same Postgres database). +Each databroker instance should have the same shared secret, but its own node id and its own storage backend. It is recommended to use the `file` storage backend. If `postgres` is used, the instances should **not** share the same Postgres database. The following settings are supported by the clustered databroker: From d7ee7064697807db875c54763eb9d2d829a3e3c1 Mon Sep 17 00:00:00 2001 From: Caleb Doxsey Date: Tue, 30 Sep 2025 09:09:51 -0600 Subject: [PATCH 4/5] update id case --- content/docs/reference/databroker.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/reference/databroker.mdx b/content/docs/reference/databroker.mdx index a5551b25b..9bcda3e6d 100644 --- a/content/docs/reference/databroker.mdx +++ b/content/docs/reference/databroker.mdx @@ -150,13 +150,13 @@ When using multiple hosts make sure to specify `target_session_attrs=read-write` As of v0.31, Pomerium supports an experimental clustered databroker. The clustered databroker consists of multiple databroker instances. One of those instances is the cluster leader and all the other instances are cluster followers. Databroker commands sent to a follower are forwarded to the leader where they are handled. In addition the clustered databroker supports automatic failure recovery via Raft leader election. -Each databroker instance should have the same shared secret, but its own node id and its own storage backend. It is recommended to use the `file` storage backend. If `postgres` is used, the instances should **not** share the same Postgres database. +Each databroker instance should have the same shared secret, but its own node ID and its own storage backend. It is recommended to use the `file` storage backend. If `postgres` is used, the instances should **not** share the same Postgres database. The following settings are supported by the clustered databroker: ### Databroker Cluster Node ID -The Databroker Cluster Node ID sets the databroker's node id. It should correspond to an entry in the Databroker Cluster Nodes option, and it should be unique for each instance of the databroker. +The Databroker Cluster Node ID sets the databroker's node ID. It should correspond to an entry in the Databroker Cluster Nodes option, and it should be unique for each instance of the databroker. From 94014e51965b04507c8c79c1a559c920c81e5049 Mon Sep 17 00:00:00 2001 From: Caleb Doxsey Date: Tue, 30 Sep 2025 09:14:12 -0600 Subject: [PATCH 5/5] updates --- content/docs/reference/databroker.mdx | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/content/docs/reference/databroker.mdx b/content/docs/reference/databroker.mdx index 9bcda3e6d..a51ff1628 100644 --- a/content/docs/reference/databroker.mdx +++ b/content/docs/reference/databroker.mdx @@ -150,7 +150,18 @@ When using multiple hosts make sure to specify `target_session_attrs=read-write` As of v0.31, Pomerium supports an experimental clustered databroker. The clustered databroker consists of multiple databroker instances. One of those instances is the cluster leader and all the other instances are cluster followers. Databroker commands sent to a follower are forwarded to the leader where they are handled. In addition the clustered databroker supports automatic failure recovery via Raft leader election. -Each databroker instance should have the same shared secret, but its own node ID and its own storage backend. It is recommended to use the `file` storage backend. If `postgres` is used, the instances should **not** share the same Postgres database. +The primary goal of clustering is to provide **resilience** and **high availability**, especially for production deployments. By running multiple databroker instances in a cluster, Pomerium can automatically recover from a single node failure in seconds, minimizing downtime and preventing data loss. + +This creates a **self-healing** system that doesn't require manual intervention to recover. It's particularly useful when using the `file` storage backend, as it replicates data across nodes to prevent data loss if a single node goes down. + +Consider enabling clustering if you are: + +- **Running a high-availability production environment** where minimizing downtime is critical. +- **Using the `file` storage backend** and need a fault-tolerant setup that can survive a single node failure. +- **Deploying across multiple regions or availability zones** to reduce latency. +- **Seeking an infrastructure-agnostic recovery solution** that works consistently across Kubernetes, VMs, and bare metal deployments. + +Each databroker instance should have the same shared secret, but its own node ID and its own storage backend. Though not recommended, if the `postgres` storage backend is used, the instances should **not** share the same Postgres database. The following settings are supported by the clustered databroker: @@ -190,6 +201,12 @@ DATABROKER_CLUSTER_NODE_ID=node-1 The Databroker Cluster Leader ID explicitly sets the leader to one of the databroker instances in the Databroker Cluster Nodes option. When this option is used, the Raft leader elector will not be used and a failure of the leader will not result in automatic failure recovery. +:::warning + +This is for advanced use cases where manual control over leadership is required. Setting a static leader disables the automatic "self-healing" capability of the cluster. + +::: + @@ -220,7 +237,7 @@ DATABROKER_CLUSTER_LEADER_ID=node-2 ### Databroker Cluster Nodes -The Databroker Cluster Nodes option defines the cluster topology of a clustered databroker. It consists of a list of node definitions, each of which has an `id`, `grpc_address` and `raft_address`. Each instance of the databroker that is part of the cluster should have the same cluster nodes definition. When using the Raft leader elector there should be an odd number of at least 3 nodes. +The Databroker Cluster Nodes option defines the cluster topology of a clustered databroker. It consists of a list of node definitions, each of which has an `id`, `grpc_address` and `raft_address`. Each instance of the databroker that is part of the cluster should have the same cluster nodes definition. To ensure quorum and prevent split-brain scenarios, a cluster using the Raft leader elector requires an odd number of nodes (a minimum of 3 is recommended).