Cluster density with Network Policies

[rsevilla87]

Description

Cluster-density is our most generalist workload and we regularly use it to measure/verify the performance of several cluster components such as the API, the SDN (both data-plane and control-plane). With the latest feedback we've got, we need to add more objects to this workload to make it more reliable.

As discussed several times, let's add some network policies to the cluster-density workload in order to make it closer to real world scenarios.

The bits proposed on this PR do the following:

• Splits the old 5 deployment into:
  • Three deployments of the server pods, based on the cloud-bulldozer's nginx image that is able to serve static content
  • Two deployments of the client pods (curl). Which have configured a readinessProbe that requests one of the 5 services backed by the server pods, and requests one of the 2 routes backed by the server serivices/pods.

We use readinessProbe rather than livenessProbe or startupProbe to prevent storms of container restarts. At the moment kube-burner waits for pods to be ready, hence all readinessProbes will have to pass prior finishing the workload.

• Add an extra route object, each client deployment use one
• In order to configure the readinessProbe against the routes, I provided kube-burner ocp the ability to discover the default ingressDomain of the cluster and expose it as a environment variable, INGRESS_DOMAIN
• Add a deny-all network policy
• Add a allow-from-ingress network policy
• Add allow-from-clients network policy

                  Client pods
              ┌────────────────────────────┐
              │                            │
              │    ┌─────┐      ┌─────┐    │
              │    │cli  │      │cli  │    │
              │    └─────┘      └─────┘    │
              │                            ├──────────────────┐
              │                            │                  │
              │     ┌─────┐     ┌─────┐    │                  │
              │     │cli  │     │cli  │    │                  │
              │     └─────┘     └─────┘    │                  │
              │                            │                  │
              └────────┬───────────────────┘                  │ReadinessProbe-2
                       │ReadinessProbe-1                      │
                       │                                      │
     ┌─────────────────▼───────────────┐                      │
     │  NetPolicy allow-from-clients   │Clients randomly picks│
     └──┬──────┬─────┬───────┬───────┬─┘ one of the 5 svc     │
        │      │     │       │       │                        │
     ┌──▼─┐ ┌──▼─┐ ┌─▼──┐  ┌─▼─┐  ┌──▼┐                       │
     │svc │ │svc │ │svc │  │svc│  │svc│                       │
     ├────┴─┴────┴─┴────┴──┴───┴──┴───┤                       └─────┐
     │                                │                             │
     │     ┌─────┐ ┌─────┐ ┌─────┐    │                        ┌────▼──────┐
     │     │nginx│ │nginx│ │nginx│    │        ┌────────────┐  │           │
     │     └─────┘ └─────┘ └─────┘    ├─────┐  │NetPolicy   ◄──┤ HAProxy   │
     │                                │Route◄──┤from-ingress◄──┤           │
     │ ┌─────┐ ┌─────┐   ┌─────┐      ├─────┘  └──┬─────────┘  │           │
     │ │nginx│ │nginx│   │nginx│      │           │            └───────────┘
     │ └─────┘ └─────┘   └─────┘      ├─────┐     │
     │                                │Route◄─────┘
     └────────────────────────────────┴─────┘    There're 2 routes
                           Nginx pods       each client deployment use one

Keep in mind that this is untested in large scale scenarios, but I'd like to have your feedback/suggestions before running it.