multus: add host checking to validation tool

[BlaineEXE]

In order to help users check that they have implemented the newly-added Multus host configuration prerequisites, add a check to the validation tool to verify connectivity.

Because users who are already running clusters with Multus enabled, add a flag that allows users to only check for host configuration prerequisites. This mode will not start the large number of clients that would normally be started because those clients could disrupt a running Rook cluster negatively.

Host checking pods require host network access. Many Kubernetes
distributions have pod security features enabled. In order to allow
non-Vanilla distros to run this tool, allow specifying a service account
that pods will run as, which can be configured by the admin to allow
test pods.

Manual validation tests:

• flakiness check still works
• expect to fail when NAD doesn't have route
• cleanup cleans up host checkers
• expect to fail when host doesn't have route
• expect to succeed when NAD and host routing is correct
• --host-check-only results in tool stopping after host check succeeds
• --host-check-only exits with early success when public-net is unset
• tool enters host check only mode when --host-check-only flag AND/OR config file config is set
• test on openshift - works, but needs SA+role+binding to allow with SCCs
  • add example resources needed for openshift?
• update documentation to mention using the validation tool
• fix KinD-based CI tests

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @BlaineEXE please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

[BlaineEXE]

I was able to get this working on openshift, but I wasn't able to define my own custom SCC. The pod was perpetually saying that it wasn't allowed by any SCC, and the custom SCC was never in the list. @subhamkrai or @Madhu-1 do you remember if you saw this issue when testing other things and how you might've resolved that?

kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: multus-validation-test
# host checker daemonset runs on host network
allowHostNetwork: true
allowedCapabilities: ["NET_BIND_SERVICE"]
runAsUser:
  type: MustRunAsNonRoot
seLinuxContext:
  type: RunAsAny
# disallow everything noncritical
allowPrivilegedContainer: false
allowHostDirVolumePlugin: false
allowHostPID: false
allowHostIPC: false
allowHostPorts: false
readOnlyRootFilesystem: true
users:
  - system:serviceaccount:openshift-storage:multus-validation-test
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus-validation-test
  namespace: openshift-storage

In the end, I was able to get this working by instead specifying a Role and RoleBinding that allowed the SA to use an existing SCC (hostnetwork-v2). I think this solution is a safe go-forward strategy, but I'm still curious why my first attempt didn't work.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: multus-validation-test
  namespace: openshift-storage
rules:
  - apiGroups:
      - security.openshift.io
    resourceNames:
      - hostnetwork-v2
    resources:
      - securitycontextconstraints
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: multus-validation-test
  namespace: openshift-storage
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: multus-validation-test
subjects:
  - kind: ServiceAccount
    name: multus-validation-test
    namespace: openshift-storage

[BlaineEXE]

All of the usage and documentation I've worked on has this tool being used interactively, especially with the config file addition, and doubly so with the openshift manifests being needed. I can't keep trying to maintain the job definition on top of the tool, so I'm removing it. I altered the docs to refer to running the tool interactively.

[travisn]

Just a few small suggestions

[travisn]

Suggested change

	[.deploy/examples/multus-validation-test-scc.yaml](https://github.com/rook/rook/blob/master/deploy/examples/multus-validation-test-scc.yaml)
	[deploy/examples/multus-validation-test-scc.yaml](https://github.com/rook/rook/blob/master/deploy/examples/multus-validation-test-scc.yaml)

[travisn]

This seems more like rbac than scc. How about naming the file multus-validation-test-rbac.yaml?

[BlaineEXE]

I thought SCC made sense because the RBAC gives access to an SCC, and SCCs are only available on openshift, implying this is an openshift-focused manifest.

[travisn]

There is an existing method k8sutil.PodsRunningWithLabel(). Would it make sense to call that method instead? Or maybe move this helper into k8sutil if it's different?

[BlaineEXE]

k8sutil.PodsRunningWithLabel() only lists pods in Running state. This method needs to list pods regardless of state.

[travisn]

just one more question...

[travisn]

How about using the k8sutil.DefaultServiceAccount to set rook-ceph-default? Or does the default need to be blank?

[BlaineEXE]

Let me think about this more and do some testing on minikube and OpenShift. This could be a good way to avoid having to use the multus-validation-test-openshift.yaml RBAC resources (or equivalent configuration on other security-constrained K8s distros), but I should verify to be sure.

Marking as draft while I look into this -- next week probably.