Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eks update-kubeconfig command does not assume role #6389

Closed
2 tasks done
carlosrodf opened this issue Sep 9, 2021 · 4 comments
Closed
2 tasks done

eks update-kubeconfig command does not assume role #6389

carlosrodf opened this issue Sep 9, 2021 · 4 comments
Assignees
Labels
closed-for-staleness eks response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@carlosrodf
Copy link

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug
When running aws eks update-kubeconfig --name <CLUSTER_NAME> --role-arn <ROLE_ARN> the command returns an access denied error:

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::*****:user/****** is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:***:*****:cluster/******

The IAM user has permissions to assume the role in question and the role has full access to EKS. I have confirmed the credentials in both user and role work by performing the steps manually:

  1. assuming the role manually through CLI
  2. exporting assumed role environment variables
  3. running aws eks update-kubeconfig... again

I report this as a bug because this command has the expected behaviour using the same IAM user and role

aws eks get-token --cluster-name <CLUSTER_NAME> --role-arn <ROLE_ARN> 

SDK version number

Platform/OS/Hardware/Device
aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off

To Reproduce (observed behavior)
Steps to reproduce the behavior

Expected behavior
The command should assume the role specified by --role-arn and update ~/.kube/config file

Logs/output

2021-09-09 15:02:33,431 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20
2021-09-09 15:02:33,432 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['eks', 'update-kubeconfig', '--name', '*****************', '--role-arn', 'arn:aws:iam::**************:role/****************', '--debug']
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fd731bd7670>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fd731d92430>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fd731db8c10>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fd731dc2d30>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fd731be90d0>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fd731d61280>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-09-09 15:02:33,438 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fd731be2310>
2021-09-09 15:02:33,438 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/awscli/data/cli.json
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fd731c90280>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fd731c90dc0>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fd731c90d30>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fd731c90ee0>
2021-09-09 15:02:33,440 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fd731c90e50>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fd731afed40>
2021-09-09 15:02:33,441 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off
2021-09-09 15:02:33,441 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['eks', 'update-kubeconfig', '--name', '*****************', '--role-arn', 'arn:aws:iam::**************:role/****************', '--debug']
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fd731bd7ca0>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fd73266df70>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fd731b44af0>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fd732669430>
2021-09-09 15:02:33,441 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fd7325e9c10>
2021-09-09 15:02:33,442 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-09-09 15:02:33,443 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fd731d61160>
2021-09-09 15:02:33,443 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fd731d8f310>
2021-09-09 15:02:33,449 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/eks/2017-11-01/service-2.json
2021-09-09 15:02:33,450 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/eks/2017-11-01/service-2.sdk-extras.json
2021-09-09 15:02:33,453 - MainThread - botocore.hooks - DEBUG - Event building-command-table.eks: calling handler <function inject_commands at 0x7fd731c6dc10>
2021-09-09 15:02:33,454 - MainThread - botocore.hooks - DEBUG - Event building-command-table.eks: calling handler <function add_waiters at 0x7fd731be2310>
2021-09-09 15:02:33,459 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/eks/2017-11-01/waiters-2.json
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event building-command-table.eks_update-kubeconfig: calling handler <function add_waiters at 0x7fd731be2310>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.kubeconfig: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.role-arn: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.dry-run: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,460 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,461 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.verbose: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,461 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.update-kubeconfig: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fd73261b970>
2021-09-09 15:02:33,461 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.update-kubeconfig.alias: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd7312ba1f0>
2021-09-09 15:02:33,461 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-09-09 15:02:33,461 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2021-09-09 15:02:33,461 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.3/dist/botocore/data/endpoints.json
2021-09-09 15:02:33,466 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fd733f42c10>
2021-09-09 15:02:33,467 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.eks: calling handler <function add_generate_presigned_url at 0x7fd733f70f70>
2021-09-09 15:02:33,471 - MainThread - botocore.endpoint - DEBUG - Setting eks timeout as (60, 60)
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.eks.DescribeCluster: calling handler <function base64_decode_input_blobs at 0x7fd731b47280>
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.eks.DescribeCluster: calling handler <function generate_idempotent_uuid at 0x7fd733f66ca0>
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event before-call.eks.DescribeCluster: calling handler <function inject_api_version_header_if_needed at 0x7fd733eec550>
2021-09-09 15:02:33,472 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeCluster) with params: {'url_path': '/clusters/*****************', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off command/eks.update-kubeconfig'}, 'body': b'', 'url': 'https://eks.********.amazonaws.com/clusters/*****************', 'context': {'client_region': '********', 'client_config': <botocore.config.Config object at 0x7fd730f2e790>, 'has_streaming_input': False, 'auth_type': None}}
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event request-created.eks.DescribeCluster: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd730f2e850>>
2021-09-09 15:02:33,472 - MainThread - botocore.hooks - DEBUG - Event choose-signer.eks.DescribeCluster: calling handler <function set_operation_specific_signer at 0x7fd733f66b80>
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/clusters/*****************

host:eks.********.amazonaws.com
x-amz-date:20210909T210233Z

host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210909T210233Z
20210909/********/eks/aws4_request
3ce24536fc7e0fb589c5154391c8d2a8997a0646e8914ff47527cc9fbba1fec0
2021-09-09 15:02:33,472 - MainThread - botocore.auth - DEBUG - Signature:
20d646a326805f289bebce51f7b259b054818128db6eef6a9cabc28f20bcbebd
2021-09-09 15:02:33,473 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://eks.********.amazonaws.com/clusters/*****************, headers={'User-Agent': b'aws-cli/2.2.3 Python/3.8.8 Linux/5.11.0-34-generic exe/x86_64.ubuntu.20 prompt/off command/eks.update-kubeconfig', 'X-Amz-Date': b'20210909T210233Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIARCH5EZTP5IDSZ5N2/20210909/********/eks/aws4_request, SignedHeaders=host;x-amz-date, Signature=20d646a326805f289bebce51f7b259b054818128db6eef6a9cabc28f20bcbebd'}>
2021-09-09 15:02:33,473 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.2.3/dist/botocore/cacert.pem
2021-09-09 15:02:33,473 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): eks.********.amazonaws.com:443
2021-09-09 15:02:33,815 - MainThread - urllib3.connectionpool - DEBUG - https://eks.********.amazonaws.com:443 "GET /clusters/***************** HTTP/1.1" 403 188
2021-09-09 15:02:33,816 - MainThread - botocore.parsers - DEBUG - Response headers: {'Date': 'Thu, 09 Sep 2021 21:02:33 GMT', 'Content-Type': 'application/json', 'Content-Length': '188', 'Connection': 'keep-alive', 'x-amzn-RequestId': '2f939d4a-ee77-4c97-8270-cd7c020d3f0a', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': '*,Authorization,Date,X-Amz-Date,X-Amz-Security-Token,X-Amz-Target,content-type,x-amz-content-sha256,x-amz-user-agent,x-amzn-platform-id,x-amzn-trace-id', 'x-amzn-ErrorType': 'AccessDeniedException', 'x-amz-apigw-id': 'FadwjGnwoAMFzZw=', 'Access-Control-Allow-Methods': 'GET,HEAD,PUT,POST,DELETE,OPTIONS', 'Access-Control-Expose-Headers': 'x-amzn-errortype,x-amzn-errormessage,x-amzn-trace-id,x-amzn-requestid,x-amz-apigw-id,date', 'X-Amzn-Trace-Id': 'Root=1-613a7669-2f77bb8127e32a5956504c13'}
2021-09-09 15:02:33,816 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"message":"User: arn:aws:iam::**************:user/************ is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:********:**************:cluster/*****************"}'
2021-09-09 15:02:33,817 - MainThread - botocore.hooks - DEBUG - Event needs-retry.eks.DescribeCluster: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd730ef72e0>>
2021-09-09 15:02:33,817 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-09-09 15:02:33,817 - MainThread - botocore.hooks - DEBUG - Event after-call.eks.DescribeCluster: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd730f2ee20>>
2021-09-09 15:02:33,818 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 459, in main
  File "awscli/clidriver.py", line 594, in __call__
  File "awscli/customizations/commands.py", line 191, in __call__
  File "awscli/customizations/eks/update_kubeconfig.py", line 122, in _run_main
  File "awscli/customizations/eks/update_kubeconfig.py", line 276, in get_cluster_entry
  File "awscli/customizations/eks/update_kubeconfig.py", line 258, in _get_cluster_description
  File "botocore/client.py", line 249, in _api_call
  File "botocore/client.py", line 568, in _make_api_call
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::**************:user/************ is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:********:**************:cluster/*****************

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::**************:user/************ is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:********:**************:cluster/*****************
@carlosrodf carlosrodf added the needs-triage This issue or PR still needs to be triaged. label Sep 9, 2021
@tim-finnigan tim-finnigan self-assigned this Sep 10, 2021
@tim-finnigan tim-finnigan added investigating This issue is being investigated and/or work is in progress to resolve the issue. eks and removed needs-triage This issue or PR still needs to be triaged. labels Sep 10, 2021
@tim-finnigan
Copy link
Contributor

Hi @carlosrodf, thanks for reaching out. Are you using a role with the IAM policy AmazonEKSClusterPolicy?

I created a Amazon EKS cluster IAM role following the steps documented here, and was able to successfully run the command aws eks update-kubeconfig --name <CLUSTER_NAME> --role-arn <ROLE_ARN>.

A similar issue was opened recently here: #5823. You could try following the steps suggested in this comment: #5823 (comment)

@tim-finnigan tim-finnigan added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Sep 10, 2021
@carlosrodf
Copy link
Author

Hi @tim-finnigan, in my case the IAM user is in a different AWS Account.

Account A:

  • IAM User

Account B:

  • IAM Role
  • EKS Cluster

The trust relationship between the role and the user works fine. I have tested assuming the role manually and it works.

@tim-finnigan
Copy link
Contributor

Hi @carlosrodf, thanks for following up. I found this blog post on enabling cross-account access to Amazon EKS cluster resources: https://aws.amazon.com/blogs/containers/enabling-cross-account-access-to-amazon-eks-cluster-resources/

Can you try following those steps and let us know if that works for you?

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Sep 11, 2021
@tim-finnigan tim-finnigan added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Sep 14, 2021
@github-actions
Copy link

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

@github-actions github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Sep 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closed-for-staleness eks response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

2 participants