fix(feg): Diameter closed connection recovery

[emakeev]

Signed-off-by: Evgeniy Makeev evgeniym@fb.com

Summary

Add connection recovery logic for externally closed diameter connection.
Current connection manager only recovers errored connections. There several scenarios when connennection can be closed 'quietly' without connection manager knowing about it - when diameter watchdog is timed out, it'll close connection unconditionally, protocol stack may closed unhealthy or idle connection, etc.

The fix is relying on diameter Close Notifier channel to detect connection closure & reconnect.

The fix can potentially resolve issue #12067

Test Plan

unit tests, TVM

Additional Information

• This change is backwards-breaking

[uri200]

is it possible this fails again and then the connection is just lost?

Meaning if for some reason there is a network outage and the server is not reacheable getDiamConnection will fail and the connection will never be created. We may end up in the same place.

Maybe we need some kind of infinite loop to keep retrying every n seconds?

if c != nil && conn != nil && c.destroyConnection(conn) {
		for {
		// if connection was closed not by connection management functions, recover it
			_, _, err := c.getDiamConnection()
			if err != nil {
				return
			}
			glog.Error("Failed to recover closed connection. Retrying")
			time.sleep(1time.Second)
		}
		```


Or maybe this loop could be at `getDiamConnection` 

[emakeev]

I think, if there is a prolonged network outage, we'll just fall back to active recovery - when the client tries to send out a request, getDiamConnection is called again and will attempt to reconnect. Putting infinite loop here could be a bit heavy, maybe a limited number of retries.

[uri200]

Note newConnection suffers of this issue. It never tries to reconnect if it fails once.
Maybe we could add a loop in that go routine and just call newConnection in this line instead of c.getDiamConnection()

http://github.com/magma/magma/blob/master/feg/gateway/diameter/connection.go#L48-L48

[uri200]

now i am just concerned about HA functionality :D

[emakeev]

HE disable will set connection to disabled...