infrastructure: tcprouter client registration with multiple tcprouter servers

[xmonader]

We have multiple tcprouter servers registered for grid.tf (or our devnet) and with the new feature to expose services using client secret , we found out that we only have the connection (between the server and the client) in only 1 tcprouter machine and in case the other ones respond it's won't work.

suggested solutions:

• floating ip: provider specific solution
• create connections to all fo the tcprouters (resolve the domain and initiate connection with them) : this one raises an issue for the best way to forward multiple connections to a single one

@zaibon @grimpy @delandtj

[grimpy]

What about to make the tcprouter client smarter

so tcprouter client connects to 1 endpoint
you register this IP for your (customer) dns

if somehow the tcprouter client disconnects from the endpoint
you reconnect to the next ip in the gw dns
and update your dns to the new endpoint

[Hamdy]

I like @grimpy solution
in addition, you may keep track in the client for latest connected IP, so it connects to it right away
after a client problem or restart. saves time and keep domain works without the need for frequent change in DNS!
also you need client to balance load in smart way, probably by using a random selection (first itme) to make sure you don't over nuke some servers , a better way is consistent hashing but I think might be complicated for this scenario here

other options if you are not choosing client side load balancing is server side load balancing
with multiple redundant load balancers with only one active at a time and holding a floating IP
in this case you could use sticky sessions to keep a client connecting to same tcprouter machine each time , but this needs some work and using some daemons like keepalived between load balancers

[zaibon]

if somehow the tcprouter client disconnects from the endpoint
you reconnect to the next ip in the gw dns
and update your dns to the new endpoint

This is not a valid solution IMO. This update of DNS record is just way to slow and requires another system to monitor this all the time.
Not really doable I think.

This tcp router client will run inside container to allow public internet access inside the container. We need something with as few management as possible. The container should just be started and be done with it. There is no way to change the configuration of the container at runtime.

The expected flow for this is the following:

1. user register its domain on a gateway and link it to a session secret
2. user deploy a container that have the tcp router client binary inside
3. the tcp router client starts and connect to the gateway(s) and identify its connection with the same secret registered in step 1.

[zaibon]

Also floating IP is not a feature we have on 0-OS so this is not something to build on top.