Add basic multi-instance e2e scale test for DNS Operator

[maleck13]

What

Currently we only test the DNSOperator either in a single cluster or at most in 2 clusters. We need to increase on this and find where there are issues beyond this scale:

Define and implement a test that will test the DNSOperators capabilities at a larger scale. A goal here would be that you could have something like

make deploy-multi INSTANCES=N

That would deploy N instances namespace scoped to the target cluster.

Ideally we would use kustomize but I am not sure it is possible to generate lots of instances with this method

We would then want something like

make test-e2e-multi INSTANCES=N

That would use deploy-multi and execute a set of tests against these deployed instances.

Proposal

Note by correct state here I am referring to the DNSRecord and possibly DNSPolicy status and also the provider zone state

• install multiple instances of the DNS operator into a single cluster that are each looking only at their own namespaces via the WATCH_NAMSPACES env var.
• Define a set of DNSRecord (based off a DNSPolicy or just install kuadrant and create a DNSPolicy) that sets up common shared dns records for a common dns name and create that DNSRecord in each of the DNS operator namespaces
• Validate that the DNSRecords get into the correct state within a defined time
• Tear down Gateways and DNSPolicies ensure correct state
• Clean up entirely and ensure correct state

Beyond a simple smoke test where the we expect the records not to conflict we also should look to cover areas such as:

• Understand where and when we begin to hit rate limits and how this presents itself and if it is clearly recognisable
• Define conflicting DNSRecords (IE a simple and loadbalanced) define and test the expected result in this case

Beyond the basic happy path defined we should also add in additional tests for common misconfiguration

• missing credentials
• bad credentials
• Bad zone definition