setup-passwords isn't figuring out how to connect to a TLS secured docker ES in 6.X

[derickson]

version: docker.elastic.co/elasticsearch/elasticsearch-platinum:6.0.0-rc1

steps to recreate:

• generate certs
• start es container for first time with TLS up for internal and external communication
• using docker exec to set default passwords, fails

docker exec <<containername>> bin/x-pack/setup-passwords auto --batch > passwords.txt

also fails when I exec into the box with an interactive terminal

error

sh-4.2$ bin/x-pack/setup-passwords auto --batch 
Exception in thread "main" java.net.SocketException: Unexpected end of file from server
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:848)
	at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.postURL(CommandLineHttpClient.java:120)
	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:273)
	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:124)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:103)

I confirmed equivalent commands work on my Mac OSX install to isolate docker as the problem

My guess is that the elasticsearch-env stuff 6.X now does to detect and inject the settings when running setup-passwords isn't figuring out what it needs to do in the Docker environment

[tvernum]

A few suggestions about what might be going wrong and how to debug.

1. It seems likely that your setup-passwords command is looking at a different configuration file than your ES server is using. That means that it cannot infer the correct URL. If you are passing ES_PATH_CONF to elasticsearch, then you need to do it for setup-passwords as well.

2. If you run with --verbose you will get more information about the URL that has been inferred.

3. You can pass a custom URL with --url

4. Or, you can pass custom ES configuration settings with -E.

5. That you are simply getting a stack trace here rather than an error message is a bug, and I'll fix it.

[jasontedor]

To add, if you’re setting up the Docker container via -e command line flags rather than static settings in a configuration file, then you have to ensure that setup-passwords is run with the same. At this point it would be better to mount a configuration file that is used (via ES_PATH_CONF or the configuration being mounted in ES_HOME/config) for both starting Elasticsearch and starting setup-passwords.

[derickson]

baby steps. Thanks for looking into this with me :)
The URL is now correct (messing with that or omitting it gets a very different error)

sh-4.2$ ./bin/x-pack/setup-passwords auto --url https://localhost:9200 -E ES_PATH_CONF=/usr/share/elasticsearch/config/ --verbose
Testing if bootstrap password is valid for https://localhost:9200/_xpack/security/_authenticate?pretty
Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)

thoughts? @tvernum

[derickson]

I'm going to give mounting an elsticsearch.yml as Jason suggests
This is as far as I could get with environment variables

sh-4.2$ ES_CONF_PATH=/usr/share/elasticsearch/config/ ./bin/x-pack/setup-passwords auto --verbose --url https://localhost:9200 \
>   -E xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/x-pack/certificates/ca/ca.crt \
>   -E xpack.security.http.ssl.verification_mode=certificate \
>   -E cluster.name=docker-cluster \
>   -E xpack.security.enabled=true \
>   -E xpack.ssl.key=/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch1/elasticsearch1.key \
>   -E xpack.ssl.certificate=/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch1/elasticsearch1.crt \
>   -E xpack.ssl.certificate_authorities=/usr/share/elasticsearch/config/x-pack/certificates/ca/ca.crt \
>   -E xpack.security.transport.ssl.enabled=true \
>   -E xpack.security.http.ssl.enabled=true 
Testing if bootstrap password is valid for https://localhost:9200/_xpack/security/_authenticate?pretty
Exception in thread "main" java.lang.IllegalArgumentException: did not find a SSLContext for [SSLConfiguration{keyConfig=[keyPath=[/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch1/elasticsearch1.key], certPaths=[/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch1/elasticsearch1.crt]], trustConfig=ca=[/usr/share/elasticsearch/config/x-pack/certificates/ca/ca.crt]], cipherSuites=[[TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[CERTIFICATE]}]
	at org.elasticsearch.xpack.ssl.SSLService.sslContextHolder(SSLService.java:320)
	at org.elasticsearch.xpack.ssl.SSLService.sslContext(SSLService.java:310)
	at org.elasticsearch.xpack.ssl.SSLService.sslSocketFactory(SSLService.java:188)
	at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.lambda$postURL$0(CommandLineHttpClient.java:95)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.elasticsearch.xpack.security.authc.esnative.tool.CommandLineHttpClient.postURL(CommandLineHttpClient.java:93)
	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:273)
	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$AutoSetup.execute(SetupPasswordTool.java:124)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool.main(SetupPasswordTool.java:103)

[derickson]

I am afraid I have hit a wall

[dliappis]

@derickson Could you try setting up passwords etc. after building your own set of images from this repo? We'll have images pushed once 6.0.0-rc2 is out.

You'd just need to:

# Ensure you are not using cached versions of 6.0.0-rc1
docker rmi docker.elastic.co/elasticsearch/elasticsearch-platinum:6.0.0-rc1 docker.elastic.co/elasticsearch/elasticsearch-basic:6.0.0-rc1 docker.elastic.co/elasticsearch/elasticsearch-oss:6.0.0-rc1 docker.elastic.co/elasticsearch/elasticsearch:6.0.0-rc1 

export ELASTIC_VERSION=6.0.0-rc1
git checkout 6.0
make

make will create updated images with the same name.

With that done, I can run the platinum flavor of the elasticseach image with something like

docker run --rm -p 19200:9200 --name testes docker.elastic.co/elasticsearch/elasticsearch-platinum:6.0.0-rc1

and then can setup passwords with:

docker exec --user 1000 -ti testes bin/x-pack/setup-passwords auto --batch

which lets me:

curl -u 'elastic:<thepassword_from_the_command_above>' localhost:19200/_cluster/health | jq .
{
  "cluster_name": "docker-cluster",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 3,
  "active_shards": 3,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 2,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 60
}

Things to note:

1. I specified --user 1000 when running the exec command as otherwise setup-passwords will run as root. (it will also run without it, but it's best to be consistent).
2. I didn't need to specify the bootstrap.password through the ELASTIC_PASSWORD env var when running the image.

[tvernum]

This error

did not find a SSLContext for [SSLConfiguration ...

occurs when you enable SSL on the HTTP interface, but don't specify a value for xpack.security.http.ssl.client_authentication

It's not a very helpful error, and I'm going to look at fixing it, but you can work around it by adding

-E xpack.security.http.ssl.client_authentication=none

[dliappis]

@derickson Did you have any success with the steps provided above? One way to pull the latest 6.0.0-rc2 snapshot is using:

curl -L snapshots.elastic.co/docker/elasticsearch-platinum-6.0.0-rc2-SNAPSHOT.tar.gz | docker load

in which case you can just run the snapshot image instead like:

docker run --rm -p 19200:9200 --name testes docker.elastic.co/elasticsearch/elasticsearch-platinum:6.0.0-rc2-SNAPSHOT

and proceed setting up the passwords using the setup-passwords tool as discussed earlier.

[derickson]

thank you @dliappis for helping me with this offline

docker exec elasticsearch1 /bin/bash -c "bin/x-pack/setup-passwords auto --batch -Expack.ssl.key=/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch1/elasticsearch1.key -Expack.ssl.certificate=/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch1/elasticsearch1.crt -Expack.ssl.certificate_authorities=/usr/share/elasticsearch/config/x-pack/certificates/ca/ca.crt -Expack.security.transport.ssl.enabled=true -Expack.security.http.ssl.enabled=true"

did the trick in RC2

I'll work on cleaning up my docker use to a elasticsearch.yml file separately. thank you all for the help