OCPBUGS-34843: UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue #75

RomanBednar · 2024-06-06T09:39:04Z

This is a cherry-pick of upstream fix kubernetes-sigs#1180 to 4.15.z.

Fix is included in upstream driver versions v1.7.1+ and 4.16 is based on upstream driver v1.7.3 so this should not regress when upgrading 4.15 -> 4.16.

Verification

Setup

Make sure we have an existing access point that does not have POSIX user set. That way driver actually loads nil posix user from AWS which is required to hit the bug:

$ aws efs describe-access-points --file-system-id fs-00d7ce85719d8f844 --output json
{
    "AccessPoints": [
        {
            "ClientToken": "console-c9a1bfd8-b40d-4f45-b50c-bdf6c6fb029f",
            "Tags": [],
            "AccessPointId": "fsap-04cfa1d5191713661",
            "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-04cfa1d5191713661",
            "FileSystemId": "fs-00d7ce85719d8f844",
            "RootDirectory": {
                "Path": "/"
            },
            "OwnerId": "269733383066",
            "LifeCycleState": "available"
        },

Storage class used:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2024-06-06T12:10:46Z"
  name: efs-sc
  resourceVersion: "66084"
  uid: eed0de64-fd0c-46bd-9f15-1587ecfac40f
mountOptions:
- tls
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: "700"
  fileSystemId: fs-00d7ce85719d8f844
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Before fix

Volume not provisioned:

oc get pvc/pvc-1
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Pending                                      efs-sc         7m34s

Controller pod panics:

$ oc -n openshift-cluster-csi-drivers logs aws-efs-csi-driver-controller-6d5c544859-5k6ft --previous | tail -n 25
Defaulted container "csi-driver" out of: csi-driver, csi-provisioner, provisioner-kube-rbac-proxy, csi-liveness-probe
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17577a0]

goroutine 1817 [running]:
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/cloud.(*cloud).ListAccessPoints(0xc000785320, {0x2029580, 0xc0004fd2c0}, {0xc00004a468, 0x14})
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/cloud/cloud.go:319 +0x320
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*GidAllocator).getUsedGids(0x6acfc0?, {0x2029580?, 0xc0004fd2c0?}, {0x202d8a0?, 0xc000785320?}, {0xc00004a468, 0x14})
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/gid_allocator.go:62 +0x65
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*GidAllocator).getNextGid(0xc0001cc158, {0x2029580, 0xc0004fd2c0}, {0x202d8a0, 0xc000785320}, {0xc00004a468, 0x14}, 0x3621e0?, 0x0?)
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/gid_allocator.go:38 +0x21c
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*Driver).CreateVolume(0xc0001cc0c0, {0x2029580, 0xc0004fd2c0}, 0xc000412460)
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/controller.go:257 +0x150e
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x2029580, 0xc0004fd2c0}, {0x1b4a380?, 0xc000412460})
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5678 +0x78
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*Driver).Run.func1({0x2029580?, 0xc0004fd2c0?}, {0x1b4a380?, 0xc000412460?}, 0xc0005aba00?, 0x19c3d80?)
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/driver.go:101 +0x42
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x1bb7f20?, 0xc0001cc0c0}, {0x2029580, 0xc0004fd2c0}, 0xc0007a0680, 0x1d19298)
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5680 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001d6000, {0x2029580, 0xc0004fd200}, {0x202f540, 0xc000009ba0}, 0xc00067e360, 0xc00065a900, 0x2f32360, 0x0)
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:1343 +0xe49
google.golang.org/grpc.(*Server).handleStream(0xc0001d6000, {0x202f540, 0xc000009ba0}, 0xc00067e360)
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:1737 +0xca6
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:986 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:997 +0x15c

After fix

PVC provisioned:

$ oc get pvc/pvc-1
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Bound    pvc-f20ba93f-b471-4dd9-9515-a8380c225a82   1Gi        RWO            efs-sc         9s

Driver controller pod logs (no panic):

$ oc -n openshift-cluster-csi-drivers logs aws-efs-csi-driver-controller-588977474f-p8ncv | tail -n 8
Defaulted container "csi-driver" out of: csi-driver, csi-provisioner, provisioner-kube-rbac-proxy, csi-liveness-probe
I0606 13:07:57.864518       1 driver.go:127] Listening for connections on address: &net.UnixAddr{Name:"/var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
W0606 13:10:29.680804       1 gid_allocator.go:88] Requested GID range (50000:7000000) exceeds EFS Access Point limit (1000) per Filesystem. Driver will use limited GID range (50000:51000)
I0606 13:10:29.680839       1 controller.go:296] Using PV name for access point directory.
I0606 13:10:29.680850       1 controller.go:303] Using /dynamic_provisioning/pvc-cb4c116c-ba12-4001-be78-fbcc06c0a767 as the access point directory.
I0606 13:12:50.882095       1 mount_linux.go:254] Cannot run systemd-run, assuming non-systemd OS
W0606 13:13:01.841162       1 gid_allocator.go:88] Requested GID range (50000:7000000) exceeds EFS Access Point limit (1000) per Filesystem. Driver will use limited GID range (50000:51000)
I0606 13:13:01.841194       1 controller.go:296] Using PV name for access point directory.
I0606 13:13:01.841207       1 controller.go:303] Using /dynamic_provisioning/pvc-f20ba93f-b471-4dd9-9515-a8380c225a82 as the access point directory.

Check EFS access points in AWS:

$ aws efs describe-access-points --file-system-id fs-00d7ce85719d8f844 --output json
{
    "AccessPoints": [
        {
            "ClientToken": "console-c9a1bfd8-b40d-4f45-b50c-bdf6c6fb029f",
            "Tags": [],
            "AccessPointId": "fsap-04cfa1d5191713661",
            "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-04cfa1d5191713661",
            "FileSystemId": "fs-00d7ce85719d8f844",
            "RootDirectory": {
                "Path": "/"
            },
            "OwnerId": "269733383066",
            "LifeCycleState": "available"
        },
        {
            "ClientToken": "pvc-f20ba93f-b471-4dd9-9515-a8380c225a82",
            "Tags": [
                {
                    "Key": "efs.csi.aws.com/cluster",
                    "Value": "true"
                },
                {
                    "Key": "kubernetes.io/cluster/rbednar-mycluster-01-qjjwv",
                    "Value": "owned"
                }
            ],
            "AccessPointId": "fsap-0e368fef6976e22b0",
            "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-0e368fef6976e22b0",
            "FileSystemId": "fs-00d7ce85719d8f844",
            "PosixUser": {
                "Uid": 50000,
                "Gid": 50000
            },
            "RootDirectory": {
                "Path": "/dynamic_provisioning/pvc-f20ba93f-b471-4dd9-9515-a8380c225a82",
                "CreationInfo": {
                    "OwnerUid": 50000,
                    "OwnerGid": 50000,
                    "Permissions": "700"
                }
            },
            "OwnerId": "269733383066",
            "LifeCycleState": "available"
        }
    ]
}

openshift-ci · 2024-06-06T09:39:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RomanBednar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [RomanBednar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2024-06-06T13:26:50Z

@RomanBednar: This pull request references Jira Issue OCPBUGS-34843, which is invalid:

expected the bug to target the "4.15.z" version, but no target version was set
release note text must be set and not match the template OR release note type must be set to "Release Note Not Required"
expected Jira Issue OCPBUGS-34843 to depend on a bug targeting a version in 4.16.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a cherry-pick of upstream fix kubernetes-sigs#1180 to 4.15.z.

Fix is included in upstream driver versions v1.7.1+ and 4.16 is based on upstream driver v1.7.3 so this should not regress when upgrading 4.15 -> 4.16.

Verification

Setup

Make sure we have an existing access point that does not have POSIX user set. That way driver actually loads nil posix user from AWS which is required to hit the bug:

$ aws efs describe-access-points --file-system-id fs-00d7ce85719d8f844 --output json
{
   "AccessPoints": [
       {
           "ClientToken": "console-c9a1bfd8-b40d-4f45-b50c-bdf6c6fb029f",
           "Tags": [],
           "AccessPointId": "fsap-04cfa1d5191713661",
           "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-04cfa1d5191713661",
           "FileSystemId": "fs-00d7ce85719d8f844",
           "RootDirectory": {
               "Path": "/"
           },
           "OwnerId": "269733383066",
           "LifeCycleState": "available"
       },

Storage class used:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 creationTimestamp: "2024-06-06T12:10:46Z"
 name: efs-sc
 resourceVersion: "66084"
 uid: eed0de64-fd0c-46bd-9f15-1587ecfac40f
mountOptions:
- tls
parameters:
 basePath: /dynamic_provisioning
 directoryPerms: "700"
 fileSystemId: fs-00d7ce85719d8f844
 provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Before fix

Volume not provisioned:

oc get pvc/pvc-1
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Pending                                      efs-sc         7m34s

Controller pod panics:

$ oc -n openshift-cluster-csi-drivers logs aws-efs-csi-driver-controller-6d5c544859-5k6ft --previous | tail -n 25
Defaulted container "csi-driver" out of: csi-driver, csi-provisioner, provisioner-kube-rbac-proxy, csi-liveness-probe
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17577a0]

goroutine 1817 [running]:
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/cloud.(*cloud).ListAccessPoints(0xc000785320, {0x2029580, 0xc0004fd2c0}, {0xc00004a468, 0x14})
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/cloud/cloud.go:319 +0x320
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*GidAllocator).getUsedGids(0x6acfc0?, {0x2029580?, 0xc0004fd2c0?}, {0x202d8a0?, 0xc000785320?}, {0xc00004a468, 0x14})
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/gid_allocator.go:62 +0x65
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*GidAllocator).getNextGid(0xc0001cc158, {0x2029580, 0xc0004fd2c0}, {0x202d8a0, 0xc000785320}, {0xc00004a468, 0x14}, 0x3621e0?, 0x0?)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/gid_allocator.go:38 +0x21c
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*Driver).CreateVolume(0xc0001cc0c0, {0x2029580, 0xc0004fd2c0}, 0xc000412460)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/controller.go:257 +0x150e
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x2029580, 0xc0004fd2c0}, {0x1b4a380?, 0xc000412460})
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5678 +0x78
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*Driver).Run.func1({0x2029580?, 0xc0004fd2c0?}, {0x1b4a380?, 0xc000412460?}, 0xc0005aba00?, 0x19c3d80?)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/driver.go:101 +0x42
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x1bb7f20?, 0xc0001cc0c0}, {0x2029580, 0xc0004fd2c0}, 0xc0007a0680, 0x1d19298)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5680 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001d6000, {0x2029580, 0xc0004fd200}, {0x202f540, 0xc000009ba0}, 0xc00067e360, 0xc00065a900, 0x2f32360, 0x0)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:1343 +0xe49
google.golang.org/grpc.(*Server).handleStream(0xc0001d6000, {0x202f540, 0xc000009ba0}, 0xc00067e360)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:1737 +0xca6
google.golang.org/grpc.(*Server).serveStreams.func1.1()
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:986 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func1
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:997 +0x15c

After fix

PVC provisioned:

$ oc get pvc/pvc-1
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Bound    pvc-f20ba93f-b471-4dd9-9515-a8380c225a82   1Gi        RWO            efs-sc         9s

Driver controller pod logs (no panic):

$ oc -n openshift-cluster-csi-drivers logs aws-efs-csi-driver-controller-588977474f-p8ncv | tail -n 8
Defaulted container "csi-driver" out of: csi-driver, csi-provisioner, provisioner-kube-rbac-proxy, csi-liveness-probe
I0606 13:07:57.864518       1 driver.go:127] Listening for connections on address: &net.UnixAddr{Name:"/var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
W0606 13:10:29.680804       1 gid_allocator.go:88] Requested GID range (50000:7000000) exceeds EFS Access Point limit (1000) per Filesystem. Driver will use limited GID range (50000:51000)
I0606 13:10:29.680839       1 controller.go:296] Using PV name for access point directory.
I0606 13:10:29.680850       1 controller.go:303] Using /dynamic_provisioning/pvc-cb4c116c-ba12-4001-be78-fbcc06c0a767 as the access point directory.
I0606 13:12:50.882095       1 mount_linux.go:254] Cannot run systemd-run, assuming non-systemd OS
W0606 13:13:01.841162       1 gid_allocator.go:88] Requested GID range (50000:7000000) exceeds EFS Access Point limit (1000) per Filesystem. Driver will use limited GID range (50000:51000)
I0606 13:13:01.841194       1 controller.go:296] Using PV name for access point directory.
I0606 13:13:01.841207       1 controller.go:303] Using /dynamic_provisioning/pvc-f20ba93f-b471-4dd9-9515-a8380c225a82 as the access point directory.

Check EFS access points in AWS:

$ aws efs describe-access-points --file-system-id fs-00d7ce85719d8f844 --output json
{
   "AccessPoints": [
       {
           "ClientToken": "console-c9a1bfd8-b40d-4f45-b50c-bdf6c6fb029f",
           "Tags": [],
           "AccessPointId": "fsap-04cfa1d5191713661",
           "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-04cfa1d5191713661",
           "FileSystemId": "fs-00d7ce85719d8f844",
           "RootDirectory": {
               "Path": "/"
           },
           "OwnerId": "269733383066",
           "LifeCycleState": "available"
       },
       {
           "ClientToken": "pvc-f20ba93f-b471-4dd9-9515-a8380c225a82",
           "Tags": [
               {
                   "Key": "efs.csi.aws.com/cluster",
                   "Value": "true"
               },
               {
                   "Key": "kubernetes.io/cluster/rbednar-mycluster-01-qjjwv",
                   "Value": "owned"
               }
           ],
           "AccessPointId": "fsap-0e368fef6976e22b0",
           "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-0e368fef6976e22b0",
           "FileSystemId": "fs-00d7ce85719d8f844",
           "PosixUser": {
               "Uid": 50000,
               "Gid": 50000
           },
           "RootDirectory": {
               "Path": "/dynamic_provisioning/pvc-f20ba93f-b471-4dd9-9515-a8380c225a82",
               "CreationInfo": {
                   "OwnerUid": 50000,
                   "OwnerGid": 50000,
                   "Permissions": "700"
               }
           },
           "OwnerId": "269733383066",
           "LifeCycleState": "available"
       }
   ]
}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

RomanBednar · 2024-06-06T13:39:38Z

/jira refresh

openshift-ci-robot · 2024-06-06T13:39:44Z

@RomanBednar: This pull request references Jira Issue OCPBUGS-34843, which is invalid:

expected dependent Jira Issue OCPBUGS-35014 to target a version in 4.16.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

RomanBednar · 2024-06-06T13:40:25Z

/jira refresh

openshift-ci-robot · 2024-06-06T13:40:33Z

@RomanBednar: This pull request references Jira Issue OCPBUGS-34843, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.z) matches configured target version for branch (4.15.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-35014 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-35014 targets the "4.16.0" version, which is one of the valid target versions: 4.16.0
bug has dependents

Requesting review from QA contact:
/cc @ropatil010

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

RomanBednar · 2024-06-06T13:41:32Z

/label backport-risk-assessed

RomanBednar · 2024-06-06T13:41:55Z

/retest-required

openshift-ci · 2024-06-06T14:42:04Z

@RomanBednar: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

RomanBednar · 2024-06-06T14:46:05Z

/cc @tsmetana FYI - backport fix tested and looks ok, reproducer is a bit trickier than adding basePath to SC.

mpatlasov · 2024-06-06T15:39:02Z

/lgtm

ropatil010 · 2024-06-06T17:08:12Z

/label cherry-pick-approved

openshift-ci-robot · 2024-06-06T17:11:55Z

@RomanBednar: Jira Issue OCPBUGS-34843: All pull requests linked via external trackers have merged:

openshift/aws-efs-csi-driver#75

Jira Issue OCPBUGS-34843 has been moved to the MODIFIED state.

In response to this:

This is a cherry-pick of upstream fix kubernetes-sigs#1180 to 4.15.z.

Fix is included in upstream driver versions v1.7.1+ and 4.16 is based on upstream driver v1.7.3 so this should not regress when upgrading 4.15 -> 4.16.

Verification

Setup

Make sure we have an existing access point that does not have POSIX user set. That way driver actually loads nil posix user from AWS which is required to hit the bug:

$ aws efs describe-access-points --file-system-id fs-00d7ce85719d8f844 --output json
{
   "AccessPoints": [
       {
           "ClientToken": "console-c9a1bfd8-b40d-4f45-b50c-bdf6c6fb029f",
           "Tags": [],
           "AccessPointId": "fsap-04cfa1d5191713661",
           "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-04cfa1d5191713661",
           "FileSystemId": "fs-00d7ce85719d8f844",
           "RootDirectory": {
               "Path": "/"
           },
           "OwnerId": "269733383066",
           "LifeCycleState": "available"
       },

Storage class used:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 creationTimestamp: "2024-06-06T12:10:46Z"
 name: efs-sc
 resourceVersion: "66084"
 uid: eed0de64-fd0c-46bd-9f15-1587ecfac40f
mountOptions:
- tls
parameters:
 basePath: /dynamic_provisioning
 directoryPerms: "700"
 fileSystemId: fs-00d7ce85719d8f844
 provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Before fix

Volume not provisioned:

oc get pvc/pvc-1
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Pending                                      efs-sc         7m34s

Controller pod panics:

$ oc -n openshift-cluster-csi-drivers logs aws-efs-csi-driver-controller-6d5c544859-5k6ft --previous | tail -n 25
Defaulted container "csi-driver" out of: csi-driver, csi-provisioner, provisioner-kube-rbac-proxy, csi-liveness-probe
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17577a0]

goroutine 1817 [running]:
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/cloud.(*cloud).ListAccessPoints(0xc000785320, {0x2029580, 0xc0004fd2c0}, {0xc00004a468, 0x14})
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/cloud/cloud.go:319 +0x320
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*GidAllocator).getUsedGids(0x6acfc0?, {0x2029580?, 0xc0004fd2c0?}, {0x202d8a0?, 0xc000785320?}, {0xc00004a468, 0x14})
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/gid_allocator.go:62 +0x65
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*GidAllocator).getNextGid(0xc0001cc158, {0x2029580, 0xc0004fd2c0}, {0x202d8a0, 0xc000785320}, {0xc00004a468, 0x14}, 0x3621e0?, 0x0?)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/gid_allocator.go:38 +0x21c
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*Driver).CreateVolume(0xc0001cc0c0, {0x2029580, 0xc0004fd2c0}, 0xc000412460)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/controller.go:257 +0x150e
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x2029580, 0xc0004fd2c0}, {0x1b4a380?, 0xc000412460})
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5678 +0x78
github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver.(*Driver).Run.func1({0x2029580?, 0xc0004fd2c0?}, {0x1b4a380?, 0xc000412460?}, 0xc0005aba00?, 0x19c3d80?)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/pkg/driver/driver.go:101 +0x42
github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x1bb7f20?, 0xc0001cc0c0}, {0x2029580, 0xc0004fd2c0}, 0xc0007a0680, 0x1d19298)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:5680 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001d6000, {0x2029580, 0xc0004fd200}, {0x202f540, 0xc000009ba0}, 0xc00067e360, 0xc00065a900, 0x2f32360, 0x0)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:1343 +0xe49
google.golang.org/grpc.(*Server).handleStream(0xc0001d6000, {0x202f540, 0xc000009ba0}, 0xc00067e360)
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:1737 +0xca6
google.golang.org/grpc.(*Server).serveStreams.func1.1()
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:986 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func1
  /go/src/github.com/kubernetes-sigs/aws-efs-csi-driver/vendor/google.golang.org/grpc/server.go:997 +0x15c

After fix

PVC provisioned:

$ oc get pvc/pvc-1
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Bound    pvc-f20ba93f-b471-4dd9-9515-a8380c225a82   1Gi        RWO            efs-sc         9s

Driver controller pod logs (no panic):

$ oc -n openshift-cluster-csi-drivers logs aws-efs-csi-driver-controller-588977474f-p8ncv | tail -n 8
Defaulted container "csi-driver" out of: csi-driver, csi-provisioner, provisioner-kube-rbac-proxy, csi-liveness-probe
I0606 13:07:57.864518       1 driver.go:127] Listening for connections on address: &net.UnixAddr{Name:"/var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
W0606 13:10:29.680804       1 gid_allocator.go:88] Requested GID range (50000:7000000) exceeds EFS Access Point limit (1000) per Filesystem. Driver will use limited GID range (50000:51000)
I0606 13:10:29.680839       1 controller.go:296] Using PV name for access point directory.
I0606 13:10:29.680850       1 controller.go:303] Using /dynamic_provisioning/pvc-cb4c116c-ba12-4001-be78-fbcc06c0a767 as the access point directory.
I0606 13:12:50.882095       1 mount_linux.go:254] Cannot run systemd-run, assuming non-systemd OS
W0606 13:13:01.841162       1 gid_allocator.go:88] Requested GID range (50000:7000000) exceeds EFS Access Point limit (1000) per Filesystem. Driver will use limited GID range (50000:51000)
I0606 13:13:01.841194       1 controller.go:296] Using PV name for access point directory.
I0606 13:13:01.841207       1 controller.go:303] Using /dynamic_provisioning/pvc-f20ba93f-b471-4dd9-9515-a8380c225a82 as the access point directory.

Check EFS access points in AWS:

$ aws efs describe-access-points --file-system-id fs-00d7ce85719d8f844 --output json
{
   "AccessPoints": [
       {
           "ClientToken": "console-c9a1bfd8-b40d-4f45-b50c-bdf6c6fb029f",
           "Tags": [],
           "AccessPointId": "fsap-04cfa1d5191713661",
           "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-04cfa1d5191713661",
           "FileSystemId": "fs-00d7ce85719d8f844",
           "RootDirectory": {
               "Path": "/"
           },
           "OwnerId": "269733383066",
           "LifeCycleState": "available"
       },
       {
           "ClientToken": "pvc-f20ba93f-b471-4dd9-9515-a8380c225a82",
           "Tags": [
               {
                   "Key": "efs.csi.aws.com/cluster",
                   "Value": "true"
               },
               {
                   "Key": "kubernetes.io/cluster/rbednar-mycluster-01-qjjwv",
                   "Value": "owned"
               }
           ],
           "AccessPointId": "fsap-0e368fef6976e22b0",
           "AccessPointArn": "arn:aws:elasticfilesystem:us-east-1:269733383066:access-point/fsap-0e368fef6976e22b0",
           "FileSystemId": "fs-00d7ce85719d8f844",
           "PosixUser": {
               "Uid": 50000,
               "Gid": 50000
           },
           "RootDirectory": {
               "Path": "/dynamic_provisioning/pvc-f20ba93f-b471-4dd9-9515-a8380c225a82",
               "CreationInfo": {
                   "OwnerUid": 50000,
                   "OwnerGid": 50000,
                   "Permissions": "700"
               }
           },
           "OwnerId": "269733383066",
           "LifeCycleState": "available"
       }
   ]
}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2024-06-06T19:09:44Z

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-aws-efs-csi-driver-container-v4.15.0-202406061806.p0.g625bd2b.assembly.stream.el8 for distgit ose-aws-efs-csi-driver.
All builds following this will include this PR.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 6, 2024

openshift-ci bot requested review from dobsonj and mpatlasov June 6, 2024 09:39

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 6, 2024

RomanBednar changed the title ~~WIP: Fixed Posixuser nil pointer dereference issue~~ OCPBUGS-34843: UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue Jun 6, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 6, 2024

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jun 6, 2024

openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jun 6, 2024

openshift-ci bot requested a review from ropatil010 June 6, 2024 13:40

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jun 6, 2024

openshift-ci bot assigned chao007, duanwei33, Phaow, radeore and ropatil010 Jun 6, 2024

UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue

ad76c6a

RomanBednar force-pushed the OCPBUGS-34843 branch from 34355cb to ad76c6a Compare June 6, 2024 14:32

openshift-ci bot assigned mpatlasov Jun 6, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 6, 2024

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jun 6, 2024

openshift-merge-bot bot merged commit 625bd2b into openshift:release-4.15 Jun 6, 2024
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-34843: UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue #75

OCPBUGS-34843: UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue #75

RomanBednar commented Jun 6, 2024 •

edited

Loading

openshift-ci bot commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

Verification

Setup

Before fix

After fix

RomanBednar commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

openshift-ci bot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

mpatlasov commented Jun 6, 2024

ropatil010 commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

Verification

Setup

Before fix

After fix

openshift-bot commented Jun 6, 2024

OCPBUGS-34843: UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue #75

OCPBUGS-34843: UPSTREAM: 1180: Fixed Posixuser nil pointer dereference issue #75

Conversation

RomanBednar commented Jun 6, 2024 • edited Loading

Verification

Setup

Before fix

After fix

openshift-ci bot commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

Verification

Setup

Before fix

After fix

RomanBednar commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

openshift-ci bot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024

mpatlasov commented Jun 6, 2024

ropatil010 commented Jun 6, 2024

openshift-ci-robot commented Jun 6, 2024

Verification

Setup

Before fix

After fix

openshift-bot commented Jun 6, 2024

RomanBednar commented Jun 6, 2024 •

edited

Loading