What happens
Restart or roll out the core, and the gateway can end up running with no WireGuard interface at all — indefinitely, until someone restarts the gateway process by hand.
The gateway removes the interface the moment the core's gRPC stream closes. When the core reconnects ten seconds later, it asks the gateway to configure the network, and that configure fails with No such device, because creating the interface is not part of that path. There is no retry. The service stays active, the log fills with statistics errors every twenty seconds, and no traffic passes for anyone on the location.
On Kubernetes this is not an edge case. The official chart sets no strategy, so the Deployment gets the Kubernetes default of RollingUpdate with maxSurge: 25%. At replicaCount: 1 that rounds up to 1 surge and down to 0 unavailable, so a new core pod always starts before the old one is terminated. Both are up briefly, the gateway accepts only one client, and the new pod is rejected with Only one client connection allowed until the old one exits — at which point there is no client left, and the interface is torn down.
We hit this three times out of three core rollouts while investigating something unrelated. Once the interface survived the rollout itself and disappeared a few minutes later, so the outcome depends on timing rather than being deterministic.
Environment
- defguard core 2.0.3
- defguard-gateway 2.0.3, kernel WireGuard (
userspace = false), installed from the release .deb on the host
- core deployed with the official chart,
replicaCount: 1, no strategy override
- Ubuntu 24.04, x86_64
clean_on_quit = false
Related evidence
The log below is a live gateway journal, hostnames stripped, otherwise verbatim. The chart observation comes from charts/defguard/templates/defguard-deployment.yaml, which sets replicas and no strategy.
The sequence
09:39:43 ERROR Only one client connection allowed
09:39:53 ERROR Only one client connection allowed
09:40:03 ERROR Only one client connection allowed
09:40:13 ERROR Only one client connection allowed
09:40:13.697 INFO Defguard Core gRPC stream has been closed
09:40:13.802 INFO Interface wg0 removed successfully
09:40:23.579 INFO Defguard Core gRPC client connected
09:40:23.579 INFO Requesting network configuration
09:40:23.633 ERROR Failed to configure: WireGuard error: Netlink error: Netlink payload error: No such device (os error 19)
After that, once every twenty seconds and nothing else, for as long as it is left alone:
ERROR Failed to retrieve WireGuard interface statistics: Netlink error: Netlink payload error: No such device (os error 19)
Note that reconnection is not the problem — the client connects successfully at 09:40:23. What fails is the configure that follows, and it is attempted exactly once.
systemctl restart defguard-gateway recovers immediately: the interface is created on startup, the core reconnects, and the full configuration applies.
Three things combine here
The interface is removed when the stream closes. This happens even with clean_on_quit = false. If an operator has explicitly asked the gateway not to clean up after itself, a dropped control connection arguably should not either.
A reconnect cannot rebuild the interface. configure_interface requires the device to exist and fails with No such device; from the observed behaviour, creation only happens on a path taken at process startup. And the failure is not retried, so the gateway sits in a dead state while otherwise healthy.
Only one client is accepted, with no handover. During a rolling update two cores briefly coexist and the new one is refused, so the moment the old one exits there is no client — which is what triggers the teardown. As a side effect, the core Deployment cannot be scaled past one replica at all: two replicas would contend for the single slot indefinitely.
Any one of these on its own would be survivable. Together they turn a routine rollout into an outage that needs manual intervention.
Why it matters
This is reachable from anything that restarts the core pod: an image bump, a GitOps sync, a node drain, an eviction, a memory limit. The gateway reports itself as active throughout, so nothing obvious points at it, and the natural reaction — restarting the gateway — makes the evidence disappear. It took us three occurrences to see the pattern.
Suggested direction.
The smallest change that removes the outage is to make a reconnect self-healing: if the interface is missing when a configuration arrives, create it rather than failing, and retry the configure instead of giving up after one attempt. That alone would reduce the whole scenario to a ten-second gap.
Beyond that, two things worth deciding: whether teardown on stream loss should respect clean_on_quit, and whether the single-client limit can allow an overlapping connection long enough for a rolling update to hand over.
What happens
Restart or roll out the core, and the gateway can end up running with no WireGuard interface at all — indefinitely, until someone restarts the gateway process by hand.
The gateway removes the interface the moment the core's gRPC stream closes. When the core reconnects ten seconds later, it asks the gateway to configure the network, and that configure fails with
No such device, because creating the interface is not part of that path. There is no retry. The service staysactive, the log fills with statistics errors every twenty seconds, and no traffic passes for anyone on the location.On Kubernetes this is not an edge case. The official chart sets no
strategy, so the Deployment gets the Kubernetes default ofRollingUpdatewithmaxSurge: 25%. AtreplicaCount: 1that rounds up to 1 surge and down to 0 unavailable, so a new core pod always starts before the old one is terminated. Both are up briefly, the gateway accepts only one client, and the new pod is rejected withOnly one client connection alloweduntil the old one exits — at which point there is no client left, and the interface is torn down.We hit this three times out of three core rollouts while investigating something unrelated. Once the interface survived the rollout itself and disappeared a few minutes later, so the outcome depends on timing rather than being deterministic.
Environment
userspace = false), installed from the release.debon the hostreplicaCount: 1, nostrategyoverrideclean_on_quit = falseRelated evidence
The log below is a live gateway journal, hostnames stripped, otherwise verbatim. The chart observation comes from
charts/defguard/templates/defguard-deployment.yaml, which setsreplicasand nostrategy.The sequence
After that, once every twenty seconds and nothing else, for as long as it is left alone:
Note that reconnection is not the problem — the client connects successfully at
09:40:23. What fails is the configure that follows, and it is attempted exactly once.systemctl restart defguard-gatewayrecovers immediately: the interface is created on startup, the core reconnects, and the full configuration applies.Three things combine here
The interface is removed when the stream closes. This happens even with
clean_on_quit = false. If an operator has explicitly asked the gateway not to clean up after itself, a dropped control connection arguably should not either.A reconnect cannot rebuild the interface.
configure_interfacerequires the device to exist and fails withNo such device; from the observed behaviour, creation only happens on a path taken at process startup. And the failure is not retried, so the gateway sits in a dead state while otherwise healthy.Only one client is accepted, with no handover. During a rolling update two cores briefly coexist and the new one is refused, so the moment the old one exits there is no client — which is what triggers the teardown. As a side effect, the core Deployment cannot be scaled past one replica at all: two replicas would contend for the single slot indefinitely.
Any one of these on its own would be survivable. Together they turn a routine rollout into an outage that needs manual intervention.
Why it matters
This is reachable from anything that restarts the core pod: an image bump, a GitOps sync, a node drain, an eviction, a memory limit. The gateway reports itself as
activethroughout, so nothing obvious points at it, and the natural reaction — restarting the gateway — makes the evidence disappear. It took us three occurrences to see the pattern.Suggested direction.
The smallest change that removes the outage is to make a reconnect self-healing: if the interface is missing when a configuration arrives, create it rather than failing, and retry the configure instead of giving up after one attempt. That alone would reduce the whole scenario to a ten-second gap.
Beyond that, two things worth deciding: whether teardown on stream loss should respect
clean_on_quit, and whether the single-client limit can allow an overlapping connection long enough for a rolling update to hand over.