AWS IRSA - Region not being passed from created Client to STS to get JWT

[snorlaX-sleeps]

Please fill out the sections below to help us address your issue

Issue description

IAM Roles for Service Accounts has been released in AWS for Kubernetes clusters running v1.13+
In the linked article, it mentions a minimum version of the Ruby SDK required to interact natively with this service (without manually making a STS call via the command line and setting env variables) is Ruby 2.11.345

The issue I am seeing is that when I pass a region parameter to instantiate a Client as part of my code, this region is not being passed to the STS call - this error occurs in all three of the clients I have tried (IAM, S3, EC2).
I assume this is something to do with the new handling of the assume_role_web_identity_credentials, released by AWS as part of this.

When using our original IAM implementation with kube2IAM, the same code does not cause any errors - only when using "IAM roles for Service Accounts"

A workaround, for anyone reading pre-fix, is to set Aws.config.update({ region: 'region' }) at the start of your code, rather than set it as part of the client, e.g ec2 = Aws::EC2::Client.new(region: 'us-west-2') - but this would not be a solution for large codebases this might affect.

Configuration options available

Error:

/usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/plugins/regional_endpoint.rb:44:in `after_initialize': missing region; use :region option or export region name to ENV['AWS_REGION'] (Aws::Errors::MissingRegionError)
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:78:in `block in after_initialize'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:77:in `each'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:77:in `after_initialize'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:21:in `initialize'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-sts/client.rb:252:in `initialize'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:99:in `new'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:55:in `initialize'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/credential_provider_chain.rb:97:in `new'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/credential_provider_chain.rb:97:in `assume_role_web_identity_credentials'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/credential_provider_chain.rb:12:in `block in resolve'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/credential_provider_chain.rb:11:in `each'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/plugins/credentials_configuration.rb:53:in `block in <class:CredentialsConfiguration>'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:70:in `call'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:213:in `block in resolve_defaults'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:57:in `each'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:57:in `each'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:212:in `resolve_defaults'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:205:in `value_at'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:189:in `block in resolve'
        from /usr/local/lib/ruby/2.5.0/set.rb:338:in `each_key'
        from /usr/local/lib/ruby/2.5.0/set.rb:338:in `each'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:189:in `resolve'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:177:in `apply_defaults'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/configuration.rb:150:in `build!'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:62:in `build_config'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:19:in `initialize'
        from /usr/local/bundle/gems/aws-sdk-ec2-1.110.0/lib/aws-sdk-ec2/client.rb:256:in `initialize'
        from /usr/local/bundle/gems/aws-sdk-core-3.68.0/lib/seahorse/client/base.rb:99:in `new'
        from ./app.rb:45:in `<main>'

This PR may be a fix - but I am not sure enough of how the gems work to make that assumption
#2090

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

Has been observed using the following gems - but may affect others
aws-sdk-s3 1.48.0
aws-sdk-ec2 1.110.0
aws-sdk-iam 1.30.0

Version of Ruby, OS environment

Ruby 2.5.1
Running on the ruby:2.5 Docker base image
Running inside a Kubernetes Cluster with v1.13 and an OIDC provider for IRSA

Code snippets / steps to reproduce

Needs to be using the IRSA features as a pod in AWS EKS or another Cluster.

Causes an issue:

require 'aws-sdk-s3'
require 'aws-sdk-ec2'
require 'aws-sdk-iam'
require 'yaml'
​
region = 'us-east-1'
​
# Logging for Docker
if ENV['DOCKER_LOGS']
  fd = IO.sysopen("/proc/1/fd/1", "w")
else
  fd = IO.sysopen("/dev/stdout", "w")
end
​
a = IO.new(fd,"w")
a.sync = true

begin
  clientEC2 = Aws::EC2::Client.new({ region: region }) # <== Causes the issue
​
  a.puts "\n Getting Worker Nodes"
  resp = clientEC2.describe_instances({
    filters: [
      {
        name: "tag:KubernetesCluster",
        values: [
          "<kluster_name>"
        ]
      },
      {
        name: "vpc-id",
        values: [
          "<vpc_id>"
        ]
      }
    ]
  })
​
  resp.reservations.each do |r|
    r.instances.each do |i|
      a.puts i.instance_id
      a.puts i.private_dns_name
      a.puts i.security_groups[0].group_name
    end
  end
​
  a.puts "\n Getting Tags for single Worker Node"
  resp = clientEC2.describe_tags({
    filters: [
      {
        name: "resource-id",
        values: [
          "<ami_id>"
        ]
      }
    ]
  })
​
  a.puts YAML.dump resp.to_h
rescue Aws::EC2::Errors::ServiceError => e
  a.puts "#{e}"
end

Working scenario:

require 'aws-sdk-s3'
require 'aws-sdk-ec2'
require 'aws-sdk-iam'
require 'yaml'
​
region = 'us-east-1'
​
# Logging for Docker
if ENV['DOCKER_LOGS']
  fd = IO.sysopen("/proc/1/fd/1", "w")
else
  fd = IO.sysopen("/dev/stdout", "w")
end
​
a = IO.new(fd,"w")
a.sync = true

Aws.config.update({ region: region })

begin
  clientEC2 = Aws::EC2::Client.new # <== No error
​
  a.puts "\n Getting Worker Nodes"
  resp = clientEC2.describe_instances({
    filters: [
      {
        name: "tag:KubernetesCluster",
        values: [
          "<kluster_name>"
        ]
      },
      {
        name: "vpc-id",
        values: [
          "<vpc_id>"
        ]
      }
    ]
  })
​
  resp.reservations.each do |r|
    r.instances.each do |i|
      a.puts i.instance_id
      a.puts i.private_dns_name
      a.puts i.security_groups[0].group_name
    end
  end
​
  a.puts "\n Getting Tags for single Worker Node"
  resp = clientEC2.describe_tags({
    filters: [
      {
        name: "resource-id",
        values: [
          "<ami_id>"
        ]
      }
    ]
  })
​
  a.puts YAML.dump resp.to_h
rescue Aws::EC2::Errors::ServiceError => e
  a.puts "#{e}"
end

[cjyclaire]

Thanks for the information! This credential provider is currently not supported in v2, the blog post is not accurate for the versioning, the feature is supported since v3, aws-sdk-core >= 3.65.0 (you can find this in core changelog as well)

v3 is modularized and compatible upgrading guide
if you are using EC2, you can have

'aws-sdk-ec2' ~> 1

make sure double checking the core version used : )

[snorlaX-sleeps]

Hi @cjyclaire - thank you for the swift response :)

I did a double check to see what error of aws-sdk-core was installed in my image, since I was hoping it was the culprit (AWS gems only)

docker run -it <image_name>:<image_version> /bin/bash
root@<id>:/usr/src/app# gem list

*** LOCAL GEMS ***

aws-eventstream (1.0.3)
aws-partitions (1.218.0)
aws-sdk-core (3.68.0)
aws-sdk-ec2 (1.110.0)
aws-sdk-iam (1.30.0)
aws-sdk-kms (1.24.0)
aws-sdk-s3 (1.48.0)
aws-sigv4 (1.1.0)

This image was only built very recently (preferring v3 modularisation over v2)
Still getting the error with the latest versions of gems - it appears that the region does not get set for the STS call (since kube2IAM calls work with the same image), so it is not getting passed / set by the <aws_service_name>::Client.new call before calling STS.

As I said, a work-around is to explicitly set Aws.config.update({ region: <region>}) - but this may cause issues if someone is trying to do something in multiple regions.

[cjyclaire]

Thanks for the information, apologies for missing the rest part when I saw the wrong version!

The question is about region setting, the default behavior when checking web assume role identities, it's checking ENVs and profile provided. one way to use the default behavior is setting ENV['AWS_REGION'], as you have mentioned, this would not be ideal for multiple clients for different regions.

While we currently support this credential provider from config, we don't take in region parameters there, this can be a feature request for supporting that, does this sound good to you?

For quick workaround, the credential interface is directly available for use, allowing client configuration.

[aackerman]

I'm running into this same issue. As part of our delivery pipeline we use scripts that handle creating resources in multiple regions, making use of ENV['AWS_REGION'] doesn't make sense for this use case where we need multiple instances of the EC2 client, one for each region.

I don't think creating an EC2 client with Aws::EC2::Client.new(region: 'us-east-1') should raise an error because you happen to be using web identity credentials.

[alextwoods]

Agreed that this shouldn't raise an error when using web identity credentials. I'll pick up adding a fix for this.

[snorlaX-sleeps]

thanks!