Refresh relays messages

[conectado]

Not necessarily the cause behind #4521 but might partially be.

Right now we get the relays credentials after we generate a connection intent, we create a new connection and we use the given TURN servers with the given credentials throughout the lifetime of the connection.

However, when there is a redploy the relays can get restarted or deleted making those credentials or whole server invalid.

The portal already have a mechanism implemented where a new relay is deployed before taking down the old one, with a new message for the gateway and client we can greatly reduce or even prevent any downtime.

With the portal broadcasting the new servers for the given connections and with a message for invalidating the old server, we can inform this to snownet and invalidate the corresponding candidates and generate new ones.

Right now the highest priority would be just to be able to inform the new TURN candidates(new servers) since with that the old ones would just timeout, without this the peer connection would be terminated invalidating all DNS mappings which would take a long time to self-heal.

[jamilbk]

We're all on the same page here. Currently wrapping up discussion w/ @thomaseizinger and @AndrewDryga on this topic and think we have a solution.

Todo

Should be backwards-compatible with existing APIs:

• Relay: Disconnect websocket on SIGTERM but keep existing data connections active
• Portal sends message that "X Relay is down, here are the N healthy ones you can use"
• Portal sends N relays in the init message in addition to request_intent (cc @conectado)

So with the above changes there are now three ways the client can receive new Relays to potentially use:

• init
• relay going down
• request_intent (receiving Relays this way is only for backwards compatibility and can go away once all clients have the above updates)

[jamilbk]

refs #4510

[conectado]

* [ ]  Portal sends N relays in the init message in addition to request_intent (cc @conectado)

@jamilbk Do we need the relays in the init message if we are already going to recieve them as part of the response to the intent?

[AndrewDryga]

• Graceful shutdown of the relay: disconnect the WS connection on SIGTERM or other signals; if there are no allocations exit immediately;
• Portal sends a message with disconnected relay ID and N relays to use; relays will be sent in init message (with backwards compatibility for now);
• Relay health checks work in two states: when relay boots it starts returning HTTP 200 right away and then connects to the websocket and if it fails to connect to the portal stops returning 200; after it was connected to a portal but then misses heartbeats for PARTITION_DURIATION stop returning HTTP 200.

[thomaseizinger]

* [ ]  Portal sends N relays in the init message in addition to request_intent (cc @conectado)

@jamilbk Do we need the relays in the init message if we are already going to recieve them as part of the response to the intent?

We also discussed removing this option entirely so first thing to do will be to remove the allow-list of TURN servers and always use all relays.

[thomaseizinger]

• when relay boots it starts returning HTTP 200 right away and then connects to the websocket

@AndrewDryga I had this differently, why return 200 initially? Shouldn't we return 400 until we are connected to the portal?

[AndrewDryga]

@thomaseizinger we need to tell instance group manager and load balancer that relay can receive traffic before the portal starts to give its address to the clients

[thomaseizinger]

Smashed out a series of PRs that move this issue forward:

• refactor(snownet): remove allow-list of STUN and TURN servers #4551
• chore(relay): perform graceful shutdown upon receiving SIGTERM #4552
• chore(relay): fail health-check with 400 on being partitioned for > 15min #4553