Elasticsearch authentication fails when "KEYSTORE_PASSWORD" is set

[kamolhasan]

Elasticsearch version (bin/elasticsearch --version):

Elasticsearch v7.14.0

Description of the problem including expected versus actual behavior:

When we try to run Elasticsearch docker image (elasticsearch:7.14.0) with both ELASTIC_PASSWORD and KEYSTORE_PASSWORD envs, the attached part of docker entrypoint script doesn't work as expected.

      if ! (run_as_other_user_if_needed echo "$KEYSTORE_PASSWORD" \
          | elasticsearch-keystore list | grep -q '^bootstrap.password$') ; then
        COMMANDS="$(printf "%s\n%s" "$KEYSTORE_PASSWORD" "$ELASTIC_PASSWORD")"
        (run_as_other_user_if_needed echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
      fi

Link: https://github.com/elastic/elasticsearch/blob/7.14/distribution/docker/src/docker/bin/docker-entrypoint.sh#L67-L71

Say, ELASTIC_PASSWORD=password and KEYSTORE_PASSWORD=key,

Output of docker-entrypoint.sh:

+ umask 0002
+ [[ eswrapper != \e\s\w\r\a\p\p\e\r ]]
+ source /usr/share/elasticsearch/bin/elasticsearch-env-from-file
++ set -e -o pipefail
++ for VAR_NAME_FILE in ELASTIC_PASSWORD_FILE KEYSTORE_PASSWORD_FILE
++ [[ -n '' ]]
++ for VAR_NAME_FILE in ELASTIC_PASSWORD_FILE KEYSTORE_PASSWORD_FILE
++ [[ -n '' ]]
+ [[ -f bin/elasticsearch-users ]]
+ [[ -n password ]]
+ [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]]
+ run_as_other_user_if_needed elasticsearch-keystore has-passwd --silent
++ id -u
+ [[ 1000 == \0 ]]
+ exec elasticsearch-keystore has-passwd --silent
+ run_as_other_user_if_needed echo key
+ elasticsearch-keystore list
+ grep -q '^bootstrap.password$'
++ id -u
+ [[ 1000 == \0 ]]
+ exec echo key
++ printf '%s\n%s' key password
+ COMMANDS='key
password'
+ run_as_other_user_if_needed echo 'key
password'
+ elasticsearch-keystore add -x bootstrap.password
++ id -u
+ [[ 1000 == \0 ]]
+ exec echo 'key
password'
++ id -u
+ [[ 1000 == \0 ]]
+ [[ -n '' ]]
+ run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch
++ id -u
+ [[ 1000 == \0 ]]
+ exec /usr/share/elasticsearch/bin/elasticsearch

Seems like the script was executed as expected. But when I try to connect to the Elasticsearch:

Client-side:

$ curl -XGET -k -u 'elastic:password' "https://localhost:9200/_cluster/health?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "Bearer realm=\"security\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Basic realm=\"security\" charset=\"UTF-8\"",
        "Bearer realm=\"security\"",
        "ApiKey"
      ]
    }
  },
  "status" : 401
}

Server-Side:

{"type": "server", "timestamp": "2021-08-12T05:45:18,307Z", "level": "INFO", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "es", "node.name": "es-0", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "TMU04RuhRUKN3oMEKybvRA", "node.id": "rVRZfOzQRdiW7CxPikHYKA"  }

N.B.:

If the bellow command: here

echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password'

Is replaced by

# remove -x flag
echo "$COMMANDS" | elasticsearch-keystore add 'bootstrap.password'

Authentication is successful. Works as expected.

Question:
- What am I missing here?
- How can I check the value of bootstrap.password stored in Keystore for debugging purposes?

[kamolhasan]

@williamrandolph Can you please take a look?

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[DJRickyB]

Just for clarification, are you setting ELASTIC_PASSWORD or ELASTICSEARCH_PASSWORD in your env?

[kamolhasan]

@DJRickyB Sorry, my bad. I'm using ELASTIC_PASSWORD in my env. Updated the description.

          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: es-auth-secret

[williamrandolph]

@kamolhasan Apologies for the delay in my response. Can you provide a full sample env file that will help me reproduce the problem? I haven't been able to reproduce this.