Make "no refresh time" configurable in `Aws::InstanceProfileCredentials`

[ybiquitous]

Describe the feature

I'd suggest that the Aws::InstanceProfileCredentials class would provide a configurable option to shorten "no refresh time" (currently implemented with the @no_refresh_until instance variable).

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb

Lines 173 to 177 in 06e3e15

	def refresh
	if @no_refresh_until && @no_refresh_until > Time.now
	warn_expired_credentials
	return
	end

Use Case

I'm using a single instance of Aws::InstanceProfileCredentials to communicate S3 in a Rails app, e.g.,

# config/initializers/aws.rb
Rails.configuration.x.aws.credentials = Aws::InstanceProfileCredentials.new
Rails.configuration.x.aws.s3_client = Aws::S3::Client.new(
  credentials: Rails.configuration.x.aws.credentials
)

In my deployment environment, an IMDS service (i.e., http://169.254.169.254) sometimes returns 503 Service Unavailable for some reason.

Once the error is raised, requests to S3 fail with 401 Unauthorized error within about 5 minutes (as Aws::InstanceProfileCredentials implements), since stale credentials are used for the S3 API. These 5-minute failures with S3 downgrade my service quality.

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb

Lines 192 to 194 in 06e3e15

	# credentials are already set, but there was an error getting new credentials
	# so don't update the credentials and use stale ones (static stability)
	@no_refresh_until = Time.now + rand(300..360)

To help understand my issue, let me share a few screenshots of APM that I'm using for the app:

• 
• 

Proposed Solution

Although there might be some approaches, I currently have the following idea to provide a new option, such as a :refresh_back Proc object:

class Aws::InstanceProfileCredentials
  def initialize(options = {})
    # ...
    @refresh_backoff = options[:refresh_backoff] || (-> { Time.now + rand(300..360) })
    @no_refresh_until = nil
    # ...
  end

  def set_refresh_interval
    @no_refresh_until = @refresh_backoff.call
  end
end

Additionally, we would need to change the message in the #warn_expired_credentials method to fit the new option:

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb

Lines 319 to 322 in 06e3e15

	def warn_expired_credentials
	warn('Attempting credential expiration extension due to a credential service availability issue. '\
	'A refresh of these credentials will be attempted again in 5 minutes.')
	end

Furthermore, it might be more helpful if this warn (i.e., Kernel#warn) were used with any specific logger object (it'd be optional, though). Since Kernel#warn outputs messages to STDOUT, they're often hard to find. For example:

def warn_expired_credentials
  logger.warn("...")
end

Other Information

This issue might be related to:

• Slow Aws::InstanceProfileCredentials.new when not running under EC2 #2617

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

aws-sdk-core 3.233.0

Environment details (OS name and version, etc.)

• OS: Linux
• Ruby: 3.4.6 (CRuby)

[mullermp]

Thanks for opening a feature request. The usage of stale credentials is intended when the service is not responding (we call this "static stability") and it is not intended to be configured.

I'd like to first drill down into why you may be getting 503 responses.

What type of credentials exist for Rails.configuration.x.aws.credentials?

Are you also using aws-sdk-rails or any of those related gems? In a previous issue we found a customer using imds credentials for every action mailer or active job event and that caused overloading of IMDS.

[ybiquitous]

@mullermp Thank you for the quick response!

I'm using the S3 client with Rails.configuration.x.aws.credentials for some controller actions that have a pretty large number of requests, so I think it's reasonable that the IMDS service may sometimes return 503.

My problem here is that the stale-credential unavailability period is too long (5 minutes). Do you have any alternatives to shortening @no_refresh_until when the IMDS is unavailable?

Also, I'm using the aws-sdk-core and aws-sdk-s3 gems.

[mullermp]

IMDS should not really return 503 as often as you are seeing. Stale credentials are used when IMDS experiences outages, but I think you are getting throttled.

The number of requests you make with s3 should not matter - IMDS credentials should be refreshed once every 10 minutes are so. Can you check logs to see how often you are attempting to refresh? You may be using multiple instances of instance profile credentials which all manage their own credentials which may repeatedly hammer the service.

[ybiquitous]

I see. Let me share my app's APM with the IMDS service (http://169.254.169.254) for the last 2 weeks:

Can you check logs to see how often you are attempting to refresh?

I'm sorry, I can't seem to get the app logs for some reason. But I guess the warning ("Attempting credential expiration extension ...") is emitted when 401 errors keep occurring after 503 errors.

[ybiquitous]

You may be using multiple instances of instance profile credentials which all manage their own credentials which may repeatedly hammer the service.

This is likely to happen. Let me check.

[ybiquitous]

You may be using multiple instances of instance profile credentials which all manage their own credentials which may repeatedly hammer the service.

This is likely to happen. Let me check.

I've confirmed a single Aws::InstanceProfileCredentials instance in the app. However, I'm using Unicorn for the Rails app with multiple server instances, so it's likely to use many different instances of Aws::InstanceProfileCredentials across worker processes in a single Unicorn process.

In this case, shortening the @no_refresh_until time wouldn't help, right? If so, to avoid the 5-minute "static stability" mode, should I adopt another credentials other than Aws::InstanceProfileCredentials as a fallback option, such as Aws::AssumeRoleCredentials?

[ybiquitous]

JFYI, let me share my app APM below. We can see the recovery from stale credentials in about 5 minutes (04:55 to 05:00).

[mullermp]

I don't think a few separate processes each calling IMDS a few times every so often will be a problem. Are you creating any clients without credentials configuration? By default, another instance of instance profile credentials would be resolved. Are you using any rails mailers with SES or active jobs with SQS? Or any underlying dependencies?

You can try globally configuring your credentials with Aws.config[:credentials] rather than per client.

[ybiquitous]

No, I'm explicitly using only Aws::InstanceProfileCredentials as a single instance for any clients in my app. And, I don't use mailers or active jobs with AWS.

If other apps (from teams other than my own) frequently submit requests to the IMDS server (http://169.254.169.254), could my app be affected and receive 503 errors? Does the IMDS endpoint depend on any requests from different instances within the same network?

[mullermp]

If there are other running apps and processes on the same host, that may be repeatedly calling IMDS, then yes it is possible you will see throttling in your app. IMDS is a link local service deployed to ec2 hosts.

[ybiquitous]

I see. Other apps are running in different hosts. So, 503 errors from my app seems to be unrelated.

So now, I understand your worries. If IMDS is dedicated to each EC2 host, I'm not sure why 503 errors so often have been raised. 🤔

[mullermp]

You might be able to use this tool to track down usage: https://github.com/aws/aws-imds-packet-analyzer

How often is IMDS being called? If it's more than 10 minutes per process then something is wrong. The credentials provider class will cache credentials and only call IMDS on refresh. Can you also check the expiration of your credentials? I've seen them anywhere from 15 minutes to 6 hours.

[ybiquitous]

Sorry for the late response. aws-imds-packet-analyzer is interested, but I cannot install it into my EC2 instances for some reason.

In my server, 20 Unicorn worker processes run, each process has a dedicated credential instance. So, I doubt whether many processes simultaneously try fetching a new token when it's expired. The 503 error rate is about 5-10%.

To reduce the number of requests to IMDS that happen simultaneously, what if we introduce jitter to the default backoff? Like this:

->(num_failures) {
  base_delay = 1.2 ** num_failures
  jitter = Kernel.rand(0..2000) / 1000.0 # 0-2 seconds
  Kernel.sleep(base_delay + jitter)
}

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb

Line 169 in 7077055

else ->(num_failures) { Kernel.sleep(1.2**num_failures) }