EFS-CSI pod impersonation implementation #710

lmouhib · 2022-06-14T17:03:36Z

Is this a bug fix or adding new feature?
new feature

What is this PR about? / Why do we need it?
Add the ability for efs-csi to impersonate pods

What testing is done?
Tested with pods that have service account annotated with a role ARN. The efs-utils assume the role annotated to the service account and is able to mount an access point that is protected with an IAM resource policy.

k8s-ci-robot · 2022-06-14T17:03:44Z

Hi @lmouhib. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wongma7 · 2022-06-14T19:45:09Z

/ok-to-test

jonathanrainer

Looks like a really good solution in general! I was working on something similar but thought it'd be better if I got behind what you were doing instead, which looks great. There are a few things to look at but feel free to push back on any of them, they're just my thoughts and I'm not expert.

I'd say a couple of general things as well as the specific points I made in the review:

It'd be worth looking into feature flagging this whole thing and setting it to false in the Helm Values. This means people have to explicitly opt-into it and will give them time to migrate, one of the most important things in this driver is backwards compatibility and if people have specific stuff setup to do with the serviceAccounts that already exist we want to give them a smooth upgrade path. I'd extend that to the new Helm Resources too.
I think it would be work extending PodAuth out and split it up into functions a bit more. This will make it easier to Unit Test because you can test one small bit at a time, while still keeping a nice public interface in the setPodCredentials function.
This definitely needs unit tests adding to it, the kubernetes library has a really nice fake library that I've added in a few PRs already. It means you can set up a fake clientset and make unit tests that react to it without a lot of effort. I'd look into that.
Does any work need to be done when NodeUnpublishVolume is called?

Let me know if some of that doesn't make sense or you need any help, it's a really great feature to see added to the driver!

cmd/main.go

jonathanrainer · 2022-06-15T05:25:22Z

pkg/driver/node.go

@@ -73,12 +93,41 @@ func (d *Driver) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolu
 	// TODO when CreateVolume is implemented, it must use the same key names
 	subpath := "/"
 	encryptInTransit := true
+	iamAuth := false


With regards to this, would it make more sense to feature flag this entire idea, then you can assume that if the user has turned the feature flag on this is what they want which gets rid of the need for this?

Ok for the feature flag, because the CSIDriver object is immutable and we need to keep backward compatibility as you mentioned. We need iamAuth there will be cases where iam auth is not needed. The Driver is used across the cluster and you might have a mix of EFS that use IAM or not. If it is set to true by default, efs-utils behavior when provided with iam mount option but there is no IAM credential available (for the pod) will try to use whatever credential available including instance profile which is not something desirable.

nit - suggest to change the var names here and below as well

pkg/driver/node.go

jonathanrainer · 2022-06-15T05:31:04Z

pkg/driver/node.go

+			result := make(map[string]interface{})
+			json.Unmarshal([]byte(volContext["csi.storage.k8s.io/serviceAccount.tokens"]), &result)
+			tokenInfo := fmt.Sprintf("%v", result["sts.amazonaws.com"].(map[string]interface{})["token"])
+


Could result be a map[string]map[string]string here? Not the prettiest data type but it avoids the use of interface{}. Further I wonder if there's a JWT library that would make this extraction a little bit easier?

The tokens are returned as an array of json string, not sure the JWT library would help here.

pkg/driver/node.go

lmouhib · 2022-06-15T17:24:21Z

@jonathanrainer Thanks for the comments, I did address some of them. Feel free to implement the unit tests, if you would like. I did not implement them yet because I did not settle on how to pass the IAM auth parameter. I believe the current way to pass it is good and keep the same structure as the tls mount option, which can be passed as a volume attribute. See example of a PV below:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: <SC>
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-xxxxx::fsap-xxxxx
    volumeAttributes:
      iamAuth: "true"

deploy/kubernetes/base/node-daemonset.yaml

Ashley-wenyizha · 2022-07-19T12:45:02Z

/ok-to-test

lmouhib · 2022-07-31T21:10:36Z

/retest

examples/kubernetes/pod_iam_impersonation/README.md

lmouhib · 2022-08-24T15:24:04Z

/retest

Ashley-wenyizha · 2022-09-15T21:33:04Z

I have pulled the code performed test locally on my side, with service account associated role have permission, mount successes, otherwise fails

Ashley-wenyizha · 2022-09-15T21:33:22Z

/lgtm

wongma7 · 2022-09-16T00:29:25Z

charts/aws-efs-csi-driver/templates/csidriver.yaml

@@ -8,3 +8,10 @@ metadata:
    "helm.sh/resource-policy": keep
 spec:
  attachRequired: false
+  {{- if .Values.iamAuth }}


maybe something a bit more descriptive for the helm value / volumeAttribute? podIAMAuth? `podIAMAuthorization? since this is b asically "mounting using the IAM authorization using the pod's IAM credentials" https://docs.aws.amazon.com/efs/latest/ug/mounting-IAM-option.html

FIxed, used podIAMAuthorization as it carry more meaning.

wongma7 · 2022-09-16T00:30:15Z

examples/kubernetes/pod_iam_impersonation/README.md

+
+This example shows how you can use the pod impersonation feature available 
+in the CSI Driver to enforce access control when mounting EFS Access point protected with EFS resource policy.
+This feature leverages IAM roles for service accounts (IRSA) and your EKS cluster must be enabled for IRSA. 


it should be stated explicitly that this feature is exclusive to EKS. right now it's implied because you need IRSA and only EKS has IRSA

Fixed, specified that it only work with EKS.

wongma7 · 2022-09-16T00:30:56Z

lgtm I just have one nitpick about the naming of the attribute which will be impossible to change later a it's basically an API ....

lmouhib · 2022-09-16T10:12:28Z

/assign @Ashley-wenyizha

Ashley-wenyizha · 2022-09-16T13:48:17Z

examples/kubernetes/pod_iam_impersonation/README.md

+    driver: efs.csi.aws.com
+    volumeHandle: [FileSystemId]::[AccessPointId]
+    volumeAttributes:
+      iamAuth: "true"


can you revise here as well?

Ashley-wenyizha · 2022-09-16T13:48:39Z

examples/kubernetes/pod_iam_impersonation/README.md

+      storage: 5Gi
+```
+
+Here it is important to set the property `iamAuth` in `volumeAttribute` to `true`. 


here as well

Ashley-wenyizha · 2022-09-16T13:48:55Z

examples/kubernetes/pod_iam_impersonation/specs/example.yaml

+    driver: efs.csi.aws.com
+    volumeHandle: fs-e8a95a42::fsap-068c22f0246419f75
+    volumeAttributes:
+      iamAuth: "true"


same as above

Ashley-wenyizha · 2022-09-16T15:19:19Z

/approve

Adding approve tag for now and let Matthew drop the lgtm tag

k8s-ci-robot · 2022-09-20T21:00:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ashley-wenyizha, lmouhib, wongma7

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Ashley-wenyizha,wongma7]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Ashley-wenyizha · 2022-09-22T18:25:30Z

/lgtm

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 14, 2022

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 14, 2022

k8s-ci-robot requested review from d-nishi and kbasv June 14, 2022 17:03

lmouhib closed this Jun 14, 2022

lmouhib reopened this Jun 14, 2022

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 14, 2022

jonathanrainer suggested changes Jun 15, 2022

View reviewed changes

jonathanrainer mentioned this pull request Jun 15, 2022

EFS CSI driver should pass the IAM flag (for IRSA) to EFS Utils for app-specific filesystem policies #280

Closed

Ashley-wenyizha reviewed Jul 19, 2022

View reviewed changes

deploy/kubernetes/base/node-daemonset.yaml Outdated Show resolved Hide resolved

lmouhib mentioned this pull request Jul 26, 2022

[EKS] [request]: EFS CSI driver should use service account role for IAM authorization aws/containers-roadmap#1003

Open

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 18, 2022

Ashley-wenyizha reviewed Aug 19, 2022

View reviewed changes

examples/kubernetes/pod_iam_impersonation/README.md Outdated Show resolved Hide resolved

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Sep 10, 2022

k8s-ci-robot assigned Ashley-wenyizha Sep 15, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 15, 2022

wongma7 reviewed Sep 16, 2022

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 16, 2022

Ashley-wenyizha reviewed Sep 16, 2022

View reviewed changes

feat: add pod impersonation feature

39373aa

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 16, 2022

wongma7 approved these changes Sep 20, 2022

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 22, 2022

k8s-ci-robot merged commit 45c5e75 into kubernetes-sigs:master Sep 22, 2022

wongma7 mentioned this pull request Sep 29, 2022

Revert "EFS-CSI pod impersonation implementation" #777

Merged

This was referenced Oct 12, 2022

1.4.2 changelog is missing #784

Closed

CHANGELOG UPDATED #789

Merged

lmouhib mentioned this pull request Jan 21, 2023

feat: Pod impersonation #885

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EFS-CSI pod impersonation implementation #710

EFS-CSI pod impersonation implementation #710

lmouhib commented Jun 14, 2022

k8s-ci-robot commented Jun 14, 2022

wongma7 commented Jun 14, 2022

jonathanrainer left a comment

jonathanrainer Jun 15, 2022

lmouhib Jun 15, 2022 •

edited

Loading

Ashley-wenyizha Sep 16, 2022 •

edited

Loading

jonathanrainer Jun 15, 2022

lmouhib Jun 15, 2022

lmouhib commented Jun 15, 2022 •

edited

Loading

Ashley-wenyizha commented Jul 19, 2022

lmouhib commented Jul 31, 2022

lmouhib commented Aug 24, 2022

Ashley-wenyizha commented Sep 15, 2022 •

edited

Loading

Ashley-wenyizha commented Sep 15, 2022

wongma7 Sep 16, 2022

lmouhib Sep 16, 2022

wongma7 Sep 16, 2022

lmouhib Sep 16, 2022

wongma7 commented Sep 16, 2022

lmouhib commented Sep 16, 2022

Ashley-wenyizha Sep 16, 2022

lmouhib Sep 16, 2022

Ashley-wenyizha Sep 16, 2022

lmouhib Sep 16, 2022

Ashley-wenyizha Sep 16, 2022

lmouhib Sep 16, 2022

Ashley-wenyizha commented Sep 16, 2022

k8s-ci-robot commented Sep 20, 2022

Ashley-wenyizha commented Sep 22, 2022

EFS-CSI pod impersonation implementation #710

EFS-CSI pod impersonation implementation #710

Conversation

lmouhib commented Jun 14, 2022

k8s-ci-robot commented Jun 14, 2022

wongma7 commented Jun 14, 2022

jonathanrainer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmouhib Jun 15, 2022 • edited Loading

Choose a reason for hiding this comment

Ashley-wenyizha Sep 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmouhib commented Jun 15, 2022 • edited Loading

Ashley-wenyizha commented Jul 19, 2022

lmouhib commented Jul 31, 2022

lmouhib commented Aug 24, 2022

Ashley-wenyizha commented Sep 15, 2022 • edited Loading

Ashley-wenyizha commented Sep 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wongma7 commented Sep 16, 2022

lmouhib commented Sep 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ashley-wenyizha commented Sep 16, 2022

k8s-ci-robot commented Sep 20, 2022

Ashley-wenyizha commented Sep 22, 2022

lmouhib Jun 15, 2022 •

edited

Loading

Ashley-wenyizha Sep 16, 2022 •

edited

Loading

lmouhib commented Jun 15, 2022 •

edited

Loading

Ashley-wenyizha commented Sep 15, 2022 •

edited

Loading