Logs full of undefined method `access_key_id'

[kichik]

Description

After upgrading from 9.5.0 to 9.10.0, our servers started using a lot more CPU and filling up their disk space. Turns out newrelic_agent.log is filled non-stop with:

WARN : Failed to create ARN: undefined method `access_key_id' for #<Aws::InstanceProfileCredentials:0x0000000006faf428>

That method doesn't exist on AWS SDK v3, but it does on v2.

I believe it's due to this change:

1329b31

Expected Behavior

I would expect no performance impact and no unsustainable amount of logging.

Steps to Reproduce

Any AWS SDK call seems to trigger it. Specifically for us we have one instance doing a lot of DynamoDB calls where we saw it affect production negatively.

Your Environment

Amazon Linux 2.

• Ruby 2.7.8
• Rails 7.0.4.2
• aws-sdk-rails 3.7
• newrelic-rpm 9.10.0

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-276779

[kaylareopelle]

I'm sorry to hear this is happening, @kichik. Thanks for bringing this to our attention.

It seems like this might be related to our new DynamoDB instrumentation introduced as part of 9.10.0.

While we look into this, I think the error should stop if you disable DynamoDB instrumentation. You can do this by updating your config/newrelic.yml file to include the following:

  instrumentation.dynamodb: disabled

Or by using an environment variable:
NEW_RELIC_INSTRUMENTATION_DYNAMODB = "disabled"

Can you let us know if that stops the problem?

[kichik]

Thanks for the quick reply @kaylareopelle. For now we downgraded to 9.9.0 and things are stable. But it's good to know DynamoDB instrumentation can be disabled. As part of this issue we have seen DynamoDB segments being 3x longer. I bet it's just the undefined method issue. But if not, it would be good to be able to disable it.

[kaylareopelle]

@kichik - Glad to hear you found a way to restore stability!

We reproduced the problem and have a solution in this branch: bugfix_aws_dynamodb_credentials_error (PR #2684)

In addition to accommodating Aws::InstanceProfileCredentials, it also introduces some performance improvements.

If you'd like to try it out, you can do so by installing newrelic_rpm from the branch:

gem 'newrelic_rpm', github: 'newrelic/newrelic-ruby-agent', branch: 'bugfix_aws_dynamodb_credentials_error'

We'll include this fix in a patch release soon, but if you decide to test it out, we'd love to hear your feedback.

[kaylareopelle]

Hi @kichik, we included #2684 in version 9.10.2, which was released yesterday. This issue should be resolved.

Thanks again for your report!

[kichik]

Thanks @kaylareopelle. The big issue seems to be resolved. But CPU is still a bit higher. Our average transaction segment time of DynamoDB went from 125ms to 150ms. CPU usage is also about 5% higher. I am not sure yet what's going on. I'll report back if I have more details.

[kaylareopelle]

Hi @kichik, thanks for the update!

If you get more details about the slowdown, we'd love to hear them. Feel free to open a new issue and we'll take a closer look 🙂

[kichik]

Thanks @kaylareopelle. Is there some guidelines on the information you need to be able to resolve such an issue? I'm not sure what to get. Right now we're thinking of rolling back again as it eats too much of our available overhead.