Aws::Signers::V4 doesn't work with Aws::RefreshingCredentials

[geissbock]

I just set up an EC2 instance running Logstash to write data to one of the brand new AWS Elasticsearch instances. All the requests need to be signed, thus the Logstash plugin uses Aws::Signers::V4 to achieve this. As I don't provide any dedicated credentials on the EC2 instance running Logstash I set up a profile that has the rights to write to Elasticsearch, which this works fine - for some hours...

Once the session token expires, it doesn't get updated in V4. I am pretty sure that the culprit is this change made to the SDK in May. Before this change, the V4 kept the Aws::CredentialProvider as an instance variable, and this has been replaced with the Aws::Credentials themselves. However, only the CredentialProvider (i.e. the Aws::InstanceProfileCredentials) is able to refresh the session token, and the Credentials will always return the same token. So after some hours you will always get the following error when using the Logstash plugin to write data to Elasticsearch:

{
:timestamp=>"2015-10-15T08:49:17.699000+0200",
:message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://...:80\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?",
:client_config=>{:hosts=>["http://...:80"], :region=>"eu-central-1", :aws_access_key_id=>nil, :aws_secret_access_key=>nil, :transport_options=>{:request=>{:open_timeout=>0, :timeout=>60}, :proxy=>nil}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::AWS, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false},
:error_message=>"[403] {\"message\":\"The security token included in the request is expired\"}",
:error_class=>"Elasticsearch::Transport::Transport::Errors::Forbidden",
:backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:135:in `__raise_transport_error'", ...],
:level=>:error
}

I spent some hours trying to write meaningful specs and/or a pull request, but I'm totally stuck. I assume that there was a good reason why keeping the CredentialProvider has been replaced with keeping the Credentials, and all I could think of right now would be a rollback :-/

Any help is appreciated - thx!

[trevorrowe]

Currently the Aws::Signers::V4 class is marked @api private and has not been considered a public interface. This has allowed us to make interface changes on the signer as needed.

A few months back the signer and credential interfaces were updated to resolve a race condition in the refreshing credential interface. The signer calls the following methods on the credentials object:

• #access_key_id
• #secret_access_key
• #session_token

Unfortunately, if a refreshing credentials object happens to refresh between invocations of those methods (some of which are called multiple times by the signer) then you make get miss-matched keys. To resolve this race condition, refreshing credentials added a #credentials method that returned the triplet of credential values for signing.

That said, I was discussing with some of the other SDK engineers yesterday. It seems like now would be an appropriate time to extract the sigv4 signer into a separate project to make it accessible, to document it, and to give it a stable interface.

As a temporary work-around, and I know this is non-optimal, you can construct a new signer and give it fresh credentials each time you sign.

[trevorrowe]

I've moved this to our feature requests backlog. That said, I consider this a high priority and hope we can get this work done quickly.

[geissbock]

Thank you for the explanation! I will forward the workaround to the guys working on the Logstash plugin.