KEYSTORE_PASSWORD & ELASTIC_PASSWORD do not work together with docker

[toughcoding]

Elasticsearch Version

8.9.0

Installed Plugins

No response

Java Version

bundled

OS Version

ubuntu:20.04 Linux 753c3b2604ff 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Problem Description

When starting Elasticsearch as a docker container with KEYSTORE_PASSWORD and ELASTIC_PASSWORD set and keystore being password secured, it does not working properly as bootstrap.password is not setup correctly.
When checking value of bootstrap.password it is returning empty string. Thus command line 54
(echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')

is not causing bootstrap.password entry being created although running these commands manually in the container are successfull.

In Contrary Running with non-encrypted keystore making bootstrap.password to be setup properly like below

docker run --rm \
-d \
-e ELASTIC_PASSWORD="123456" \
docker.elastic.co/elasticsearch/elasticsearch:8.9.0

Steps to Reproduce

Create container only to make encrypted keystore

docker run --rm \
--name elk \
-d \
-v esconf:/usr/share/elasticsearch/config \
docker.elastic.co/elasticsearch/elasticsearch:8.9.0

Run command to setup password for already created keystore setting password as 123456
docker exec -it elk elasticsearch-keystore passwd

docker exec -it elk elasticsearch-keystore list  
Enter password for the elasticsearch keystore : 
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

Stop container
docker stop elk

Start another container with password variables

docker run --rm \
--name elk \
-d \
-v esconf:/usr/share/elasticsearch/config \
-e ELASTIC_PASSWORD="987654" \
-e KEYSTORE_PASSWORD="123456" \
docker.elastic.co/elasticsearch/elasticsearch:8.9.0

Checking if bootstrap password entry got created

docker exec -it elk elasticsearch-keystore list
Enter password for the elasticsearch keystore : 
bootstrap.password
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password

Run command to display bootstrap.password that normally should be equal to ELASTIC_PASSWORD
docker exec -it elk elasticsearch-keystore show bootstrap.password

BUT it is returning empty string thus bug.

Logs (if relevant)

Beginning of logs after starting second container. Script asking twice for keystore password.
docker run --rm
--name elk
-v esconf:/usr/share/elasticsearch/config
-e ELASTIC_PASSWORD="123456"
-e KEYSTORE_PASSWORD="123456"
docker.elastic.co/elasticsearch/elasticsearch:8.9.0
Enter password for the elasticsearch keystore : Enter password for the elasticsearch keystore : {"@timestamp":"2023-08-01T19:08:33.302Z", "log.level": "INFO",

[toughcoding]

There was related discussion for it on forum.
https://discuss.elastic.co/t/keystore-password-file/339627/4

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

@elastic/es-security I'm having issues even trying to reproduce this. If I create a password-protected keystore, then bind mount it into the container and attempt to start ES it blows up during security auto-configuration. Is this a known issue?

Enter password for the elasticsearch keystore : Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:874)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
	at java.base/java.nio.file.Files.move(Files.java:1429)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:514)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
	at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:582)
	at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:165)
	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:86)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.Command.main(Command.java:50)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
	Suppressed: java.lang.UnsupportedOperationException: Unsupported copy option
		at java.base/sun.nio.fs.UnixFileSystem$Flags.fromMoveOptions(UnixFileSystem.java:500)
		at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:861)
		at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
		at java.base/java.nio.file.Files.move(Files.java:1429)
		at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:587)

[ywangd]

I think this is related to #76124

The issue is not with docker. It's a bug for adding string value to a password protected ES keystore. Assuming the keystore password is 123456, it can be reproduced with

COMMANDS="$(printf "%s\n%s" "123456" "abcdefg")"
echo "$COMMANDS" | ./bin/elasticsearch-keystore add -x 'foo'

# The following command shows empty for `foo`
./bin/elasticsearch-keystore show foo

Basically the terminal reader for keystore password eagerly reads everything and leaves nothing for the actual content.
I think it needs attention from @elastic/es-core-infra
Also ping @albertzaharovits since he was the author of the linked PR.

[toughcoding]

@elastic/es-security I'm having issues even trying to reproduce this. If I create a password-protected keystore, then bind mount it into the container and attempt to start ES it blows up during security auto-configuration. Is this a known issue?

Enter password for the elasticsearch keystore : Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:874)
	at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
	at java.base/java.nio.file.Files.move(Files.java:1429)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:514)
	at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
	at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:582)
	at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:165)
	at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:86)
	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
	at org.elasticsearch.cli.Command.main(Command.java:50)
	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
	Suppressed: java.lang.UnsupportedOperationException: Unsupported copy option
		at java.base/sun.nio.fs.UnixFileSystem$Flags.fromMoveOptions(UnixFileSystem.java:500)
		at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:861)
		at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
		at java.base/java.nio.file.Files.move(Files.java:1429)
		at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:587)

To make it work like you wish I am running first container with mounted volume into config and generate password protected keystore. Then secondly I am stopping that container and starting new container with mounted config catalog not just keystore file. So I am using volumes therefore not bind mounts. Elastic team do not recommend to bind mount keystore file mentioning it can cause issues.

[toughcoding]

I think this is related to #76124

The issue is not with docker. It's a bug for adding string value to a password protected ES keystore. Assuming the keystore password is 123456, it can be reproduced with

COMMANDS="$(printf "%s\n%s" "123456" "abcdefg")"
echo "$COMMANDS" | ./bin/elasticsearch-keystore add -x 'foo'

# The following command shows empty for `foo`
./bin/elasticsearch-keystore show foo

Basically the terminal reader for keystore password eagerly reads everything and leaves nothing for the actual content. I think it needs attention from @elastic/es-core-infra Also ping @albertzaharovits since he was the author of the linked PR.

Yes it is not due to docker itself but on high level running docker does not work as you might expect from documentation.

[rjernst]

I was positive we had keystore tests for this exact scenario. However, upon closer inspection I found that the tests are bogus for a variety of reasons. Unfortunately there are a couple places where we use buffered readers which break this. Fixing it is non trivial; the internal Terminal abstraction needs more work to cleanup how we handle raw input.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[da1910]

Its been a little while but I think I'm seeing this same issue when trying to use a precreated keystore, we're providing the password to the keystore in the KEYSTORE_PASSWORD environment variable, but the entrypoint continues to prompt for a password.

If I exec into a running container I can see the variable is set, and if I then run /bin/elasticsearch and enter the keystore password things proceed. But it's not respecting the environment variable as far as I can see. Is there a workaround in the mean time?

[toughcoding]

@da1910 You can check my article about this. Maybe you will find some inspiration from commands. Otherwise let's wait for Elastic to come back with suggestions.