Regression in 7.9: AWS signing appears to be broken

[myronmarston]

We are using the elasticsearch gem from an AWS lambda that talks to an AWS elasticsearch cluster. The security of our AWS elasticsearch cluster requires all requests to be signed with AWS sig4 using the AWS credentials provided by lambda execution role (which has been granted access the the elasticsearch cluster in its IAM policy).

In our AWS lambda, we create our client like so:

require 'elasticsearch'
require 'faraday_middleware/aws_sigv4'

elastic_client = Elasticsearch::Client.new(url: ENV.fetch("ELASTICSEARCH_URL"), logger: nil) do |f|
  f.request :aws_sigv4,
    service: 'es',
    region: ENV.fetch('AWS_REGION'), # assumes the lambda and ES domain live in the same region.
    access_key_id: ENV.fetch('AWS_ACCESS_KEY_ID'),
    secret_access_key: ENV.fetch('AWS_SECRET_ACCESS_KEY'),
    session_token: ENV['AWS_SESSION_TOKEN'] # optional
end

When we were using version 7.8.1 of the elasticsearch gem, this worked just fine. After upgrading the elasticsearch gem to 7.9.0, we immediately started to get errors on every request:

Elasticsearch::Transport::Transport::Errors::Forbidden
User: arn:aws:sts::<redacted_aws_account_id>:assumed-role/<redacted_lambda_name>_<redacted_aws_region>_execution_role/<redacted_lambda_name> is not authorized to perform: es:ESHttpPost

Here's the bits of the stack trace from the elasticsearch gems:

elasticsearch-transport-7.9.0/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error'
elasticsearch-transport-7.9.0/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request'
elasticsearch-transport-7.9.0/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'
elasticsearch-transport-7.9.0/lib/elasticsearch/transport/client.rb:176:in `perform_request'
elasticsearch-api-7.9.0/lib/elasticsearch/api/actions/msearch.rb:89:in `msearch'

When we reverted back to 7.8.1 the problem went away and our requests to our AWS Elasticsearch cluster started to succeed again. So, it appears there is some regression in 7.9.0 that somehow breaks the AWS Sig4 signing in such a way that our requests are rejected when they should not be.

I dug through the Changelog and didn't see anything that appeared to be related to this but I may have missed it.

[picandocodigo]

Hi @myronmarston,
Looking into this, the only changes from the transport layer from 7.8.1 to 7.9 are the documented in the Changelog (06ffd03) and some styling changes which don't affect functionality.

What did change though is msearch will use POST as the request method since it has a body. It used to use GET since it was the first method in the spec, but I added a check to use POST when there's a body to conform with proper HTTP. Elasticsearch supports both so it should work either way. From the error you posted, is it possible that the user might need extra permissions to perform a POST request?

User: arn:aws:sts::(...) is not authorized to perform: es:ESHttpPost

Can you try checking for this and letting me know? Thank you!

[myronmarston]

Thanks for the quick response @picandocodigo! You're absolutely right -- we have allowed our IAM user to perform the es:ESHttpGet and es:ESHttpHead actions but had intentionally not allowed it to perform the es:ESHttpPost, es:ESHttpPut and es:ESHttpDelete actions. The idea is that we have some lambdas that should only be allowed to read data from Elasticsearch (and should never be allowed to write data). We have other lambdas that should only be allowed to write data to Elasticsearch, and other lambdas that should be allowed to read and write data. It makes our security model a bit stronger if the lambdas are only granted the ability to read data if they should be allowed to, and only granted the ability to write data if they should be allowed to. We used Get/Head as the list of "read" actions and Post/Put/Delete as the list of "write" actions but it sounds like that's not going to work if we upgrade.

I always thought it was funny that Elasticsearch used a GET with a request body (it's quite nonstandard...) but in practice it was quite nice for our AWS security model :). I'm not sure if there's a simple way we can prevent our lambda from being able to write data if we have to grant it the es:ESHttpPost action for it to be able to use msearch.

Given that the GET w/ body approach worked fine for years, and that it allows this simple security model, would you be open to making it configurable? Then we could configure out client to continue using GET for msearch.

Thanks!

[picandocodigo]

@myronmarston Yes, we agreed on the clients team to default to a POST request when there's a body for most clients, since it's what makes the most sense. For now we won't change it back, but there is a workaround. You can build the request from the client manually instead of using elastic_client.msearch and use a GET request for msearch:

method = Elasticsearch::API::HTTP_GET
path = "#{index}/_msearch"
elastic_client.perform_request(method, path, params, payload, headers)

Check out msearch's source code to make sure the body is in the expected format and check how we build the msearch request.

[myronmarston]

Thanks, I actually found a cleaner way to do this using a Faraday middleware. In case it helps anyone, here it is:

require 'elasticsearch'

module MyApplication
  module Elastic
    class ClientFactory
      def self.new_client(*args)
        Elasticsearch::Client.new(*args) do |conn|
          conn.use MSearchUsingGetInsteadOfPost
          yield conn if block_given?
        end
      end

      class MSearchUsingGetInsteadOfPost
        def initialize(app)
          @app = app
        end

        def call(env)
          env.method = :get if env.url.path.end_with?("_msearch")
          @app.call(env)
        end
      end
    end
  end
end

....and then I use MyApplication::Elastic::ClientFactory.new_client in my application to create the client.

Thanks again for the help!