instance profile credentials expiring

[chrono]

The plugin (I'm using kinesis_producer) does not seem fetch fresh tokens from the metadata service regularly.

2016-03-27 05:20:10 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2016-03-27 05:29:17 +0000 error_class="StandardError" error="{:attempts=>[{:delay=>112, :dura
tion=>6, :success=>false, :error_code=>\"400\", :error_message=>\"{\\\"__type\\\":\\\"ExpiredTokenException\\\",\\\"message\\\":\\\"The security token included in the request is
expired\\\"}\"}], :success=>false}" plugin_id="object:3ff76076a31c"

The credentials seem to be fetched once at plugin startup in https://github.com/awslabs/aws-fluent-plugin-kinesis/blob/v1.0.0/lib/fluent/plugin/kinesis_helper/credentials.rb#L47. The name of the method default_credentials_provider is suggesting that it returns a provider, instead it returns credentials.

The credentials are then passed in to the client which has a loop to refresh the credentials https://github.com/awslabs/aws-fluent-plugin-kinesis/blob/v1.0.0/lib/kinesis_producer/daemon.rb#L183-L192. After ~6 hours, the credentials are no longer valid and the producer stalls.

Shouldn't this use the provider instead and dereference the credentials property regularly?

[edited to replace master with v1.0.0 in the file links]

[riywo]

Hi @chrono ,

I'll look into this issue. KPL itself has credentials_refresh_delay option, so I think it should refresh credentials automatically. But I'll double check it.

[riywo]

OK, I think it'll be a bug. I'll fix it.

[riywo]

Hi @chrono ,

I released v1.0.1 which fixes this problem. Please try this version. Thank you for reporting!

After confirming your situation is resolved, I'll yank v1.0.0 from rubygems since this is a huge bug.

[josqu4red]

I had the same problem, but using kinesis_streams output.
No more ExpiredTokenExceptions since upgraded to 1.0.1 yesterday.
Thanks!

[chrono]

Yup it has been working for me for the last few days on 1.0.1. So I can ditch my custom patched 1.0.0 now :)

[chrono]

Thanks for the quick turnaround!

[riywo]

@josqu4red @chrono

Great to hear from you! I'm so sorry for inconvenience. If you have any issue, feel free to submit an issue on this repository.

[vishalmamidi]

fluent/fluent-plugin-opensearch#24