Summary
In collector mode (gnmic collect), the cluster leader's HTTP clients that call other members' REST APIs (target assignment, unassignment, target deletion) are plain &http.Client{} instances that never load the clustering.tls configuration. As a result:
A private-CA server certificate on api-server.tls breaks target dispatch with x509: certificate signed by unknown authority.
mTLS (api-server.tls.client-auth: require-verify) is impossible: the dispatch client presents no client certificate, so the leader gets remote error: tls: bad certificate calling members — including itself.
This works correctly in subscribe mode, where clustering.tls was wired up by #544 (v0.39.0, for #529). Collector mode (added in v0.43.0) has a separate cluster implementation under pkg/collector/managers/cluster/ that never got the same treatment — the code still carries // TODO: markers at the relevant client.
Version
main @ 4c6c11a (2026-07-09), also reproduced on v0.46.0.
Reproduce
3-instance gnmic collect cluster + Consul locker. Enable TLS on the API with a private CA:
api-server:
tls:
ca-file: /certs/ca.pem # implies client-auth: require-verify (mTLS)
cert-file: /certs/server.pem
key-file: /certs/server-key.pem
clustering:
cluster-name: c1
instance-name: ${INSTANCE}
service-address: ${ADDR}
tls: # documented on the HA page, but ignored here
ca-file: /certs/ca.pem
cert-file: /certs/client.pem
key-file: /certs/client-key.pem
locker: { type: consul, address: consul:8500 }
Add a target via the API. The leader elects fine, then fails to dispatch:
level=INFO msg="assigning target" target=X assignee=gnmic-0
http: TLS handshake error from : remote error: tls: bad certificate
level=ERROR msg="failed to push assignments" error="Post "https://gnmic-0...:7890/api/v1/assignments\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
The target is never assigned; no telemetry. (Server-only TLS with a publicly trusted cert and no client-auth would connect, but private-CA and mTLS — the normal on-prem setup — cannot.)
Root cause (main @ 4c6c11a)
The clustering TLS config is fully parsed and even env-expanded, but never used to build the dispatch clients:
Parsed: pkg/config/clustering.go:37 — TLS *types.TLSConfig, populated at pkg/config/clustering.go:57-61.
Env-expanded (proof it's processed in collector mode): pkg/collector/env/env.go:18-21 expands clusteringConfig.TLS.{CaFile,CertFile,KeyFile}.
Never consumed — the two outbound inter-member clients are hardcoded plain clients:
pkg/collector/managers/cluster/cluster_manager.go:69 — apiClient: &http.Client{Timeout: 10 * time.Second}, // TODO:
pkg/collector/managers/cluster/assigner.go:48 — client: &http.Client{Timeout: 10 * time.Second}
grep -rn 'Clustering.TLS' pkg/collector/ returns nothing — the collector package never reads it.
Meanwhile members register __protocol=https whenever api-server.tls != nil (cluster_manager.go:240-244), so the assigner correctly builds https:// URLs (assigner.go:61,105) — it just dials them with a TLS-unaware client.
Expected behavior
Collector-mode dispatch clients should honor clustering.tls exactly as subscribe mode does, so private-CA server certs verify and client certs are presented for mTLS.
Suggested fix
Mirror the reference implementation App.createAPIClient() at pkg/app/clustering.go:704-737 (which builds the client via utils.NewTLSConfig(CaFile, CertFile, KeyFile, "", SkipVerify, false)), and inject the resulting *http.Client into both NewClusterManager (replacing the // TODO: client at cluster_manager.go:69) and NewAssigner (assigner.go:44-52). Both already have access to the clustering config (ClusterManager.clusteringConfig), so the client can be constructed once from clusteringConfig.TLS and shared.
Impact / workaround
On-prem clusters (private CA, mTLS policy) cannot secure the collector API end-to-end today. Current workarounds: (a) api-server.tls with cert/key only — no ca-file, no client-auth (encryption + server-auth only; caution: setting ca-file silently defaults client-auth to require-verify and breaks the cluster), plus SSL_CERT_FILE=<ca.pem> in the process environment so the dispatch client trusts the internal CA, and network-layer authn (firewall / k8s NetworkPolicy / service-mesh mTLS); or (b) use subscribe mode (where clustering.tls works) with a target loader instead of collector mode.
Summary
In collector mode (gnmic collect), the cluster leader's HTTP clients that call other members' REST APIs (target assignment, unassignment, target deletion) are plain &http.Client{} instances that never load the clustering.tls configuration. As a result:
A private-CA server certificate on api-server.tls breaks target dispatch with x509: certificate signed by unknown authority.
mTLS (api-server.tls.client-auth: require-verify) is impossible: the dispatch client presents no client certificate, so the leader gets remote error: tls: bad certificate calling members — including itself.
This works correctly in subscribe mode, where clustering.tls was wired up by #544 (v0.39.0, for #529). Collector mode (added in v0.43.0) has a separate cluster implementation under pkg/collector/managers/cluster/ that never got the same treatment — the code still carries // TODO: markers at the relevant client.
Version
main @ 4c6c11a (2026-07-09), also reproduced on v0.46.0.
Reproduce
3-instance gnmic collect cluster + Consul locker. Enable TLS on the API with a private CA:
api-server:
tls:
ca-file: /certs/ca.pem # implies client-auth: require-verify (mTLS)
cert-file: /certs/server.pem
key-file: /certs/server-key.pem
clustering:
cluster-name: c1
instance-name: ${INSTANCE}
service-address: ${ADDR}
tls: # documented on the HA page, but ignored here
ca-file: /certs/ca.pem
cert-file: /certs/client.pem
key-file: /certs/client-key.pem
locker: { type: consul, address: consul:8500 }
Add a target via the API. The leader elects fine, then fails to dispatch:
level=INFO msg="assigning target" target=X assignee=gnmic-0
http: TLS handshake error from : remote error: tls: bad certificate
level=ERROR msg="failed to push assignments" error="Post "https://gnmic-0...:7890/api/v1/assignments\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
The target is never assigned; no telemetry. (Server-only TLS with a publicly trusted cert and no client-auth would connect, but private-CA and mTLS — the normal on-prem setup — cannot.)
Root cause (main @ 4c6c11a)
The clustering TLS config is fully parsed and even env-expanded, but never used to build the dispatch clients:
Parsed: pkg/config/clustering.go:37 — TLS *types.TLSConfig, populated at pkg/config/clustering.go:57-61.
Env-expanded (proof it's processed in collector mode): pkg/collector/env/env.go:18-21 expands clusteringConfig.TLS.{CaFile,CertFile,KeyFile}.
Never consumed — the two outbound inter-member clients are hardcoded plain clients:
pkg/collector/managers/cluster/cluster_manager.go:69 — apiClient: &http.Client{Timeout: 10 * time.Second}, // TODO:
pkg/collector/managers/cluster/assigner.go:48 — client: &http.Client{Timeout: 10 * time.Second}
grep -rn 'Clustering.TLS' pkg/collector/ returns nothing — the collector package never reads it.
Meanwhile members register __protocol=https whenever api-server.tls != nil (cluster_manager.go:240-244), so the assigner correctly builds https:// URLs (assigner.go:61,105) — it just dials them with a TLS-unaware client.
Expected behavior
Collector-mode dispatch clients should honor clustering.tls exactly as subscribe mode does, so private-CA server certs verify and client certs are presented for mTLS.
Suggested fix
Mirror the reference implementation App.createAPIClient() at pkg/app/clustering.go:704-737 (which builds the client via utils.NewTLSConfig(CaFile, CertFile, KeyFile, "", SkipVerify, false)), and inject the resulting *http.Client into both NewClusterManager (replacing the // TODO: client at cluster_manager.go:69) and NewAssigner (assigner.go:44-52). Both already have access to the clustering config (ClusterManager.clusteringConfig), so the client can be constructed once from clusteringConfig.TLS and shared.
Impact / workaround
On-prem clusters (private CA, mTLS policy) cannot secure the collector API end-to-end today. Current workarounds: (a) api-server.tls with cert/key only — no ca-file, no client-auth (encryption + server-auth only; caution: setting ca-file silently defaults client-auth to require-verify and breaks the cluster), plus SSL_CERT_FILE=<ca.pem> in the process environment so the dispatch client trusts the internal CA, and network-layer authn (firewall / k8s NetworkPolicy / service-mesh mTLS); or (b) use subscribe mode (where clustering.tls works) with a target loader instead of collector mode.