Standalone gateway returns out-of-date topology when brokers go away

[jwulf]

Part of a series of "you're doing it wrong" tests to determine the behavior of the system when misconfigured or in a failed state.

How does the standalone gateway behave / respond when the broker behind it goes away?

[jwulf]

Topology query gives a cached result. Attempts to deploy time out:

zeebe-docker-compose/bin on  0.18 [!] at ☸️  ZeebeCluster 
➜ ./zbctl.darwin status --address 127.0.0.1:26500
Cluster size: 3
Partitions count: 2
Replication factor: 3
Brokers:
  Broker 0 - 192.168.128.3:26501
    Partition 1 : Follower
    Partition 2 : Follower
  Broker 1 - 192.168.128.4:26501
    Partition 1 : Follower
    Partition 2 : Follower
  Broker 2 - 192.168.128.5:26501
    Partition 1 : Leader
    Partition 2 : Leader

zeebe-docker-compose/bin on  0.18 [!] at ☸️  ZeebeCluster 
➜ ./zbctl.darwin deploy ~/Downloads/diagram_1.bpmn
Error: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL
Usage:
  zbctl deploy <workflowPath> [flags]

Flags:
  -h, --help   help for deploy

Global Flags:
      --address string   Specify the Zeebe addressFlag

[Zelldon]

The gateway will not request the broker for the topology. The Gateway is part of the Atomix cluster. It should take some time that the topology is adjusted, since the configuration changes/node events are spread over SWIM.

[jwulf]

After 7 minutes it still hasn't updated. Is that expected?

[Zelldon]

Nope, then it is a bug I would say.

[npepinpe]

Is this still occurring? We patched Atomix so that the nodes periodically sync against each other to propagate a consistent view (every 10 seconds by default). There is still a SPoF since the Gateway initially only knows a single broker, but this is only an issue if that broker isn't there when the Gateway initially connects.

[npepinpe]

Can we check if this still occurs, and verify that this is only reporting the wrong topology and not actually using it?

[npepinpe]

Blocked by #4554

[npepinpe]

So I thought it wasn't occurring anymore, as we have many tests indirectly covering this - turns out, while we test when there are leaders, we almost never test what happens if almost all brokers leave, e.g. 2 out of 3. Turns out the gateway doesn't remove leaders, just replaces them, so yes, we sometimes return the wrong topology. This can affect that the gateway will keep trying to ping the old leader even if it isn't leader, and thus return the wrong error.