Write e2e test to verify full stack is deployed

[jsanda]

I would like to see an e2e integration test that deploys the full stack. We The test should verify that each of the components is initialized. To the extent possible this should be done through the status of each of the objects.

The CassandraDatacenter object for example has the cassandraOperatorProgress field which is set to true when the operator is done doing work. There is also condition named Ready whose status will be set to true when all the Cassandra pods have reached the ready state.

The Reaper object simply has a ready field in its status that is set to true when Reaper reaches the ready state.

We want perform similar verification through the statuses of Prometheus, Grafana, and Stargate as well.

Before checking each of those objects, we should probably first check that each of the operators is successfully deployed. If cass-operator fails to deploy for example, there is not point to try and check the CassandraDatacenter.

For Prometheus, I think we additionally should verify that targets have been discovered. We have example of this already being done in simple_integration_test.go.

For Grafana if possible I think we should verify that the Prometheus data source is registered and also check that the dashboards are imported.

We have people developing on both Mac and on Linux. The test should be able to run on both platforms. Furthermore, the test should not only be able to run on kind but other distros as well, e.g., k3d, GKE, etc. I am not saying we need to run the test against N different distros, but should be a general enough test that it should run on any (or most) distros provided it meets the minimim Kubernetes version requirements.

simple_integration_test.go deploys Traefik and verifies ingress as well. If using Traefik will make it a lot more difficult to target different k8s distros, then I think we should consider tests with and without ingress.

[jsanda]

simple_integration_test.go sets up the cluster and configures traefik as described in the docs here.

Here is the traefik chart configuration:

providers:
  kubernetesCRD:
    namespaces:
      - default
      - traefik
  kubernetesIngress:
    namespaces:
      - default
      - traefik

ports:
  traefik:
    expose: true
    nodePort: 32090
  web:
    nodePort: 32080
  websecure:
    nodePort: 32443
  cassandra:
    port: 9042
    nodePort: 32091
  cassandrasecure:
    port: 9142
    nodePort: 32092

service:
  type: NodePort

If we want to use this on a managed k8s cluster like GKE, we will need to specify firewall rules to open the ports. I don't think the tests themselves need to do that configuration. I think it would be great though if we could have a script that will make the necessary changes to configure the environment. I currently do most of my development and testing with GKE so I would like some helper scripts for GKE. At some point we would want similar setup scripts for EKS and any other cloud providers we need to support.