[Delta server] Properly close watches when a stream is closed

[valerian-roche]

We noticed on our environments that control-planes getting no activity outside of serving xds through delta-ads to a constant nb of pods had an ever-growing consumption of memory
Investigating further we noticed that:

• this memory increase correlated to a nb of goroutine increase (even if the nb of clients is constant)
• the increase occur each time we restart those client pods (done on purpose to test some edge cases)
• the increase is two goroutines per pod, which makes sense as those use lds and cds as dynamic resources and do not trigger rds, eds or rtds watches

Taking a goroutine dump we identified that leaked goroutines are from https://github.com/envoyproxy/go-control-plane/blob/main/pkg/server/delta/v3/server.go#L180
Those goroutines will only be gc-ed once the channel has either returned a response or been closed. This closure is expected to happen in watch.Cancel(), but in the case of a node getting removed, we are actually calling the potentially set cancel function attached to the watch, but we do not close this channel

Disclaimer: we use a slightly different fork of the repository. While I do not believe it impacts here, we are not using vanilla control-plane today

goroutine count with/without the fix over time:

[alecholmez]

Thanks for this change! Glad we can cleanup this leak