fix(kuma-cp) Clear snapshots from cache on disconnect

[lahabana]

Previously we were pushing an empty snapshot on the cache when a dp was disconnecting.
This was causing the dp to get an empty snapshot after a restart.
This would cause listeners to disappear for a short time and causing the traffic going through the dp to fail.
We now clear the snapshot from the dp to avoid this behaviour.

Fix #2171

Testing

• Unit tests
• E2E tests
• Manual testing on Universal
• Manual testing on Kubernetes

[jakubdyszkiewicz]

Ok, I think this change makes sense.

The only problem is that I believe this change will make SDS tests really flaky. I remember that with a clear snapshot it was a huge problem. But I'd rather have a flaky test than a problem on prod.

Let's just keep in mind that we may need to revisit SDS tests after merging it.

[lahabana]

@jakubdyszkiewicz was it e2e tests or unit tests that were flaky? I can try to look at the flakiness if you tell me where to start :)

[jakubdyszkiewicz]

@lahabana pkg/sds/server/v3/server_test.go this one. I remember I was only able to reproduce it when I was hammering my VM with yes > /dev/null & and running tests in a loop.

[lahabana]

Ok thanks I'll try to repro! Should I merge this anyway and potentially fix the test in a follow up PR?

[jakubdyszkiewicz]

yes, definitely. Go ahead and merge it (squash and merge 🙏 )

[lahabana]

yes, definitely. Go ahead and merge it (squash and merge 🙏 )

Can an admin make squash and merge the default as well :) (And maybe disable the other options in the github repo settings)?

[jakubdyszkiewicz]

it should be by default 🤔 or maybe it's only on my profile

[lahabana]

Hey @jakubdyszkiewicz I can't manage to make the tests flaky. I've done 154 attempts of the tests (ginkgo -v -untilItFails -race pkg/sds/server/...) with a docker image with 0.5 cpu and yes > /dev/null running and never got any failure.
Don't you think you change on ed84c9c might have fixed the flakiness?