Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add "deletecollection" verb to "persistentvolumeclaims" Kubernetes RBAC permission #17

Conversation

alopezsanchez
Copy link
Contributor

Description

Hi 👋🏻

We are getting an error when executing an EMR job on a virtual cluster (EKS) using the role and the rolebinding created with the virtual-cluster submodule.
The rolebinding between the role and the service account is correct.

Command:

aws emr-containers start-job-run \
   --virtual-cluster-id "virtual-cluster-id" \
   --name=test_emr \
   --region "xxxxx" \
   --execution-role-arn "arn:aws:iam::0000000000:role/execution-role" \
   --release-label emr-7.0.0-latest \
   --job-driver '{
       "sparkSubmitJobDriver":{
               "entryPoint": "s3://path_to_jar/jar_name.jar",
               "entryPointArguments":["argument_name_1","argument_1"],
               "sparkSubmitParameters": "--class MapperEntrypoint --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=1 --conf spark.driver.cores=1"
       }
   }' \
   --configuration-overrides '{
        "applicationConfiguration": [
          {
            "classification": "spark-defaults",
            "properties": {
              "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
              "spark.sql.catalog.spark_catalog":"org.apache.spark.sql.delta.catalog.DeltaCatalog",
              "spark.jars.packages": "com.amazonaws:aws-java-sdk-s3:1.12.401,com.amazonaws:aws-java-sdk-secretsmanager:1.12.401,org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.2,io.delta:delta-spark_2.12:3.0.0"
            }
          }
        ],
       "monitoringConfiguration": {
            "persistentAppUI": "ENABLED",
            "s3MonitoringConfiguration": {
               "logUri": "s3://path_to_logs"
            }
       }
   }'

Error:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default.svc/api/v1/namespaces/emr-production/persistentvolumeclaims?labelSelector=spark-app-selector%3Dspark-000000033c4hs4rueii.
Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. persistentvolumeclaims is forbidden: User "system:serviceaccount:emr-production:emr-containers-sa-spark-driver-702351764633-2paf8mkf46wdp1726c2315tio0bk0ult5j2i3dp304fcwq27hb2touevq1l3xggvbl1" cannot deletecollection resource "persistentvolumeclaims" in API group "" in the namespace "emr-production".

Motivation and Context

We are not sure if we are missing something, so feel free to close it if you think we are doing something wrong.

How Has This Been Tested?

  • I have executed pre-commit run -a on my pull request

We are getting this error when executing an EMR job using the role and the role
bindings installed in this module.

Command:
```
aws emr-containers start-job-run \
   --virtual-cluster-id "virtual-cluster-id" \
   --name=test_emr \
   --region "xxxxx" \
   --execution-role-arn "arn:aws:iam::0000000000:role/execution-role" \
   --release-label emr-7.0.0-latest \
   --job-driver '{
       "sparkSubmitJobDriver":{
               "entryPoint": "s3://path_to_jar/jar_name.jar",
               "entryPointArguments":["argument_name_1","argument_1"],
               "sparkSubmitParameters": "--class MapperEntrypoint --conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=1 --conf spark.driver.cores=1"
       }
   }' \
   --configuration-overrides '{
        "applicationConfiguration": [
          {
            "classification": "spark-defaults",
            "properties": {
              "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
              "spark.sql.catalog.spark_catalog":"org.apache.spark.sql.delta.catalog.DeltaCatalog",
              "spark.jars.packages": "com.amazonaws:aws-java-sdk-s3:1.12.401,com.amazonaws:aws-java-sdk-secretsmanager:1.12.401,org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.2,io.delta:delta-spark_2.12:3.0.0"
            }
          }
        ],
       "monitoringConfiguration": {
            "persistentAppUI": "ENABLED",
            "s3MonitoringConfiguration": {
               "logUri": "s3://path_to_logs"
            }
       }
   }'
```

Error:
```
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://kubernetes.default.svc/api/v1/namespaces/emr-production/persistentvolumeclaims?labelSelector=spark-app-selector%3Dspark-000000033c4hs4rueii.
Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. persistentvolumeclaims is forbidden: User "system:serviceaccount:emr-production:emr-containers-sa-spark-driver-702351764633-2paf8mkf46wdp1726c2315tio0bk0ult5j2i3dp304fcwq27hb2touevq1l3xggvbl1" cannot deletecollection resource "persistentvolumeclaims" in API group "" in the namespace "emr-production".
```
@alopezsanchez alopezsanchez changed the title fix(virtual-cluster): add missing Kubernetes RBAC permission fix(virtual-cluster): Add missing Kubernetes RBAC permission Feb 8, 2024
@bryantbiggs bryantbiggs changed the title fix(virtual-cluster): Add missing Kubernetes RBAC permission fix: Add "deletecollection" Kubernetes RBAC permission Feb 8, 2024
@bryantbiggs bryantbiggs changed the title fix: Add "deletecollection" Kubernetes RBAC permission fix: Add "deletecollection" verb to "persistentvolumeclaims" Kubernetes RBAC permission Feb 8, 2024
Copy link
Member

@bryantbiggs bryantbiggs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@bryantbiggs bryantbiggs merged commit 668f09b into terraform-aws-modules:master Feb 8, 2024
16 of 17 checks passed
antonbabenko pushed a commit that referenced this pull request Feb 8, 2024
### [1.2.1](v1.2.0...v1.2.1) (2024-02-08)

### Bug Fixes

* Add `"deletecollection"` verb to `"persistentvolumeclaims"` Kubernetes RBAC permission ([#17](#17)) ([668f09b](668f09b))
@antonbabenko
Copy link
Member

This PR is included in version 1.2.1 🎉

Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants