Submariner in different AZs?

[m4ce]

We tried running submariner in our cluster which spans three different AZs on AWS. It looks like the route agents fail to add the route to the destination cluster CIDRs when using as a default gateway a local node that sits in a different AZ. This is to be expected since you cannot route from one AZ via another in AWS since they are not directly attached. How to solve this?

[Oats87]

We are looking at a solution to fix this, like using VXLAN to pass local cluster traffic instead of using ip routing.

[m4ce]

Eventually, we decided to run a submariner per AZ and a broker per AZ. So the cluster id concept is mapped to the AZ. As we had to run the submariner agents in different namespaces, we had to patch the code as the namespace (submariner) was hard-coded. We also had to change the route agents DaemonSet to have a specific node selector based on the AZ (cluster id). The second cluster also has the same number of AZs and they use the same brokers. We now have it working between two clusters and multiple AZs and routing works as expected.

[dimaunx]

#154