Race condition on cluster creation/deletion causes unexpected behaviour

[jgwest]

On cluster creation/deletion, I have seen cases where:

• If a new Argo CD cluster is created, and the ApplicationSet is reconciled before the 'new cluster event' is received by the Argo CD DB cluster lister, then a new Application is failed to be created for the cluster.
• If a new Argo CD cluster is deleted, and the ApplicationSet is reconciled before the 'cluster deleted event' is received by the Argo CD DB cluster lister, then the Application fails to be deleted.

This can be reproduced by running the E2E tests that @OmerKahani implemented as part of #148 (eg the tests that creates a new cluster secret).

I've got a PR for this, but I'm targeting this for the next milestone because I think the fix would be too invasive at this late stage for the 0.1.0 release, because it is partially mitigated by #170, and because it should occur infrequently in real world settings.

I was reminded of this issue that I opened, which might be related: argoproj/argo-cd#4755

• I could see this as being the Argo CD equivalent of this ApplicationSet issue

Log

When running make test-e2e with Omer's E2E PR, it's reproducible about 20% of the time

=== RUN   TestSimpleClusterGeneratorAddingCluster
time="2021-03-25T12:32:25-04:00" level=info msg="mkdir -p /tmp/applicationset-e2e" dir= execID=ey9ig
time="2021-03-25T12:32:25-04:00" level=debug duration=2.24523ms execID=ey9ig
time="2021-03-25T12:32:25-04:00" level=info msg="clean state" duration=46.269361ms id= name=TestSimpleClusterGeneratorAddingCluster password=password username=admin
time="2021-03-25T12:32:26-04:00" level=info msg="action: creating cluster Secret 'my-secret'"
time="2021-03-25T12:32:26-04:00" level=info msg="expectation succeeded: no error and output contained ''"
time="2021-03-25T12:32:26-04:00" level=info msg="action: creating ApplicationSet 'simple-cluster-generator'"
time="2021-03-25T12:32:26-04:00" level=info msg="expectation succeeded: no error and output contained ''"
time="2021-03-25T12:32:26-04:00" level=info msg="expectation pending: missing app 'cluster1-guestbook'"
time="2021-03-25T12:32:29-04:00" level=info msg="expectation succeeded: all apps successfully found"
time="2021-03-25T12:32:29-04:00" level=info msg="action: creating cluster Secret 'my-secret2'"
time="2021-03-25T12:32:29-04:00" level=info msg="expectation succeeded: no error and output contained ''"
time="2021-03-25T12:32:29-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:32-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:35-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:38-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:41-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:45-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:48-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:51-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:54-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
time="2021-03-25T12:32:57-04:00" level=info msg="expectation pending: missing app 'cluster2-guestbook'"
    cluster_e2e_test.go:299: timeout waiting for: missing app 'cluster2-guestbook'
--- FAIL: TestSimpleClusterGeneratorAddingCluster (34.83s)