-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multus: add host checking to validation tool #14230
multus: add host checking to validation tool #14230
Conversation
2b5e387
to
0355394
Compare
f96dbc6
to
ea13a6a
Compare
This pull request has merge conflicts that must be resolved before it can be merged. @BlaineEXE please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork |
639a122
to
8fc06e7
Compare
I was able to get this working on openshift, but I wasn't able to define my own custom SCC. The pod was perpetually saying that it wasn't allowed by any SCC, and the custom SCC was never in the list. @subhamkrai or @Madhu-1 do you remember if you saw this issue when testing other things and how you might've resolved that?
In the end, I was able to get this working by instead specifying a Role and RoleBinding that allowed the SA to
|
4433dac
to
9e899a5
Compare
9e899a5
to
1f79e2b
Compare
622c6e3
to
12cc170
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All of the usage and documentation I've worked on has this tool being used interactively, especially with the config file addition, and doubly so with the openshift manifests being needed. I can't keep trying to maintain the job definition on top of the tool, so I'm removing it. I altered the docs to refer to running the tool interactively.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few small suggestions
the tool's `serviceAccountName` config option or `--service-account-name` CLI flag to instruct | ||
the tool to run using a particular ServiceAccount in order to allow necessary permissions. | ||
An example compatible with openshift is provided in the Rook repository at | ||
[.deploy/examples/multus-validation-test-scc.yaml](https://github.com/rook/rook/blob/master/deploy/examples/multus-validation-test-scc.yaml) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[.deploy/examples/multus-validation-test-scc.yaml](https://github.com/rook/rook/blob/master/deploy/examples/multus-validation-test-scc.yaml) | |
[deploy/examples/multus-validation-test-scc.yaml](https://github.com/rook/rook/blob/master/deploy/examples/multus-validation-test-scc.yaml) |
@@ -0,0 +1,38 @@ | |||
# ServiceAccount and RBAC to support running multus validation test on OpenShift |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems more like rbac than scc. How about naming the file multus-validation-test-rbac.yaml
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought SCC made sense because the RBAC gives access to an SCC, and SCCs are only available on openshift, implying this is an openshift-focused manifest.
@@ -349,16 +370,13 @@ func (vt *ValidationTest) numClientsReady(ctx context.Context, expectedNumPods i | |||
return numReady, nil | |||
} | |||
|
|||
func (vt *ValidationTest) getClientPods(ctx context.Context, expectedNumPods int) (*core.PodList, error) { | |||
func (vt *ValidationTest) getPodsWithLabel(ctx context.Context, label string) (*core.PodList, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is an existing method k8sutil.PodsRunningWithLabel()
. Would it make sense to call that method instead? Or maybe move this helper into k8sutil
if it's different?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
k8sutil.PodsRunningWithLabel()
only lists pods in Running state. This method needs to list pods regardless of state.
e508cdb
to
aa71688
Compare
aa71688
to
b67e28a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just one more question...
pkg/daemon/multus/config.go
Outdated
@@ -38,6 +38,8 @@ var ( | |||
var ( | |||
DefaultValidationNamespace = "rook-ceph" | |||
|
|||
DefaultServiceAccountName = "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about using the k8sutil.DefaultServiceAccount
to set rook-ceph-default
? Or does the default need to be blank?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me think about this more and do some testing on minikube and OpenShift. This could be a good way to avoid having to use the multus-validation-test-openshift.yaml
RBAC resources (or equivalent configuration on other security-constrained K8s distros), but I should verify to be sure.
Marking as draft while I look into this -- next week probably.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to use the rook-ceph-system
service account on openshift (from operator hub), and it isn't working again just as here: #14230 (comment)
Manual error. Using the rook-ceph-system
SA is working!
In order to help users check that they have implemented the newly-added Multus host configuration prerequisites, add a check to the validation tool to verify connectivity. Because users who are already running clusters with Multus enabled, add a flag that allows users to only check for host configuration prerequisites. This mode will not start the large number of clients that would normally be started because those clients could disrupt a running Rook cluster negatively. Host checking pods require host network access. Many Kubernetes distributions have pod security features enabled. In order to allow non-Vanilla distros to run this tool, allow specifying a service account that pods will run as, which can be configured by the admin to allow test pods. Signed-off-by: Blaine Gardner <blaine.gardner@ibm.com>
b67e28a
to
33f5407
Compare
multus: add host checking to validation tool (backport #14230)
In order to help users check that they have implemented the newly-added Multus host configuration prerequisites, add a check to the validation tool to verify connectivity.
Because users who are already running clusters with Multus enabled, add a flag that allows users to only check for host configuration prerequisites. This mode will not start the large number of clients that would normally be started because those clients could disrupt a running Rook cluster negatively.
Host checking pods require host network access. Many Kubernetes
distributions have pod security features enabled. In order to allow
non-Vanilla distros to run this tool, allow specifying a service account
that pods will run as, which can be configured by the admin to allow
test pods.
Manual validation tests:
cleanup
cleans up host checkersChecklist: