Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to start Elastic Search using IBM JRE #32146

Closed
dubauski opened this issue Jul 18, 2018 · 15 comments

Comments

@dubauski
Copy link

commented Jul 18, 2018

Elasticsearch version (bin/elasticsearch --version): 6.3.1

Plugins installed: none

JVM version (java -version):
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 8.0.5.11 - pxa6480sr5fp11-20180326_01(SR5 FP11))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64 Compressed References 20180309_380776 (JIT enabled, AOT enabled)
OpenJ9 - 49fcaf39
OMR - 5cbbadf
IBM - 4453dac)
JCL - 20180319_01 based on Oracle jdk8u161-b12

OS version (uname -a if on a Unix-like system):
elasticsearch@a3821331cba5:/elasticsearch-6.3.1/bin$ uname -a
Linux a3821331cba5 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
bin/elasticsearch-keystore create
or
bin/elasticsearch

fail to run with the following errors

elasticsearch@a3821331cba5:/elasticsearch-6.3.1/bin$ ./elasticsearch
Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.security.spec.InvalidKeySpecException: Could not generate secret key
Likely root cause: java.lang.RuntimeException: Error deriving PBKDF2 keys
at com.ibm.crypto.provider.PBKDF2KeyImpl.a(Unknown Source)
at com.ibm.crypto.provider.PBKDF2KeyImpl.(Unknown Source)
at com.ibm.crypto.provider.bm.engineGenerateSecret(Unknown Source)
at javax.crypto.SecretKeyFactory.generateSecret(Unknown Source)
at org.elasticsearch.common.settings.KeyStoreWrapper.createCipher(KeyStoreWrapper.java:289)
at org.elasticsearch.common.settings.KeyStoreWrapper.encrypt(KeyStoreWrapper.java:361)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:473)
at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:234)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Refer to the log for complete error details.

Steps to reproduce:

  1. bin/elasticsearch-keystore create
    OR
  2. bin/elasticsearch

Provide logs (if relevant):
elasticsearch@a3821331cba5:/elasticsearch-6.3.1/bin$ ./elasticsearch-keystore create
Exception in thread "main" java.security.spec.InvalidKeySpecException: Could not generate secret key
at javax.crypto.SecretKeyFactory.generateSecret(Unknown Source)
at org.elasticsearch.common.settings.KeyStoreWrapper.createCipher(KeyStoreWrapper.java:289)
at org.elasticsearch.common.settings.KeyStoreWrapper.encrypt(KeyStoreWrapper.java:361)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:473)
at org.elasticsearch.common.settings.CreateKeyStoreCommand.execute(CreateKeyStoreCommand.java:58)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:79)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:41)
Caused by: java.lang.RuntimeException: Error deriving PBKDF2 keys
at com.ibm.crypto.provider.PBKDF2KeyImpl.a(Unknown Source)
at com.ibm.crypto.provider.PBKDF2KeyImpl.(Unknown Source)
at com.ibm.crypto.provider.bm.engineGenerateSecret(Unknown Source)
... 11 more

@dubauski

This comment has been minimized.

Copy link
Author

commented Jul 18, 2018

I have tried running Elastic Search 6.3.1 with OpenJDK and it ran fine. The problem is that our environment mandates use of IBM JRE 8. Is there a way to workaround the problem?

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Jul 18, 2018

Only the OpenJDK and OracleJDK are supported: https://www.elastic.co/support/matrix#matrix_jvm

@nik9000 nik9000 closed this Jul 18, 2018
@dubauski

This comment has been minimized.

Copy link
Author

commented Jul 18, 2018

Thank you for quick reply @nik9000. Is this something new for ES 6.3.x? We were able to run ES 6.2.2 with IBM JRE successfully despite information on the https://www.elastic.co/support/matrix#matrix_jvm

@nik9000

This comment has been minimized.

Copy link
Contributor

commented Jul 18, 2018

I don't know that we've ever supported the IBM JDK. We don't test it because we don't support it so I'm not surprised that we broke support for it in a minor release. Sorry!

@bizybot

This comment has been minimized.

Copy link
Contributor

commented Jul 18, 2018

@dubauski
The keystore format has been changed to make it FIPS compliant. As we use empty passwords, this fails on IBM JDK which does not allow empty passwords.
https://stackoverflow.com/questions/43294652/secretkeyfactory-generatesecret-dies-with-invalidkeyspecexception-on-ibm-java
As Nik has already pointed we do not support IBM JDK, I am just providing a workaround which may work. Just deploy an old version of ES and create the keystore. Copy that keystore in the config directory. Hope this helps.

@dubauski

This comment has been minimized.

Copy link
Author

commented Jul 18, 2018

I appreciate all the timely responses to this issue.

@bizybot thank you for the suggested workaround - I'll give it a try.

@dubauski

This comment has been minimized.

Copy link
Author

commented Jul 18, 2018

Workaround update.

  1. I tried creating the keystore using 6.2.2 ES which by the way launched successfully with IBM JRE. The "elasticsearch-keystore create" command failed with the same error complaining about inability to generate secret key.
  2. I then produced the keystore file using 6.3.1 ES instance running on a different system with non IBM JRE. I then placed the elasticsearch-keystore file under config folder and tried to run elasticsearch. Unfortunately that still didn't help.

Here is the full output:

elasticsearch@1251b82f3210:/elasticsearch-6.3.1$ bin/elasticsearch
Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.security.spec.InvalidKeySpecException: Could not generate secret key
Likely root cause: java.lang.RuntimeException: Error deriving PBKDF2 keys
at com.ibm.crypto.provider.PBKDF2KeyImpl.a(Unknown Source)
at com.ibm.crypto.provider.PBKDF2KeyImpl.(Unknown Source)
at com.ibm.crypto.provider.bm.engineGenerateSecret(Unknown Source)
at javax.crypto.SecretKeyFactory.generateSecret(Unknown Source)
at org.elasticsearch.common.settings.KeyStoreWrapper.createCipher(KeyStoreWrapper.java:289)
at org.elasticsearch.common.settings.KeyStoreWrapper.decrypt(KeyStoreWrapper.java:336)
at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:237)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Refer to the log for complete error details.
elasticsearch@1251b82f3210:/elasticsearch-6.3.1$ ls config
elasticsearch.keystore jvm.options role_mapping.yml users
elasticsearch.yml log4j2.properties roles.yml users_roles

@bizybot did I miss a property in elasticsearch.yml?

elasticsearch.yml.txt

@boboci9

This comment has been minimized.

Copy link

commented Sep 11, 2018

Any news about this? I have the same error.

@dubauski

This comment has been minimized.

Copy link
Author

commented Sep 11, 2018

Few months ago I opened RFE against IBM JRE to handle empty passwords better. Unfortunately it appears that using OpenJDK is the only option for ES 6.3.1+.

@bellatrix1001

This comment has been minimized.

Copy link

commented Oct 22, 2018

Few months ago I opened RFE against IBM JRE to handle empty passwords better. Unfortunately it appears that using OpenJDK is the only option for ES 6.3.1+.

@dubauski : any luck figuring out a workaround for this? I recently tried ES 6.4.2 on IBM JRE and despite Elastic's comments in their own forums that they will abandon PBKDF2 keys - still getting the same error. Thanks for your help!

@dubauski

This comment has been minimized.

Copy link
Author

commented Oct 22, 2018

@bellatrix1001 no luck. I tried to address this problem by opening PMR and then RFE against IBM JRE but unfortunately after 3 months of waiting IBM JRE team decided to not do anything about it.

If our team has to move forward with ES upgrade from 6.2.2 to later one I can only see few options:

  1. use non-IBM JRE
  2. introduce non-empty password into our ES setup (which could be a costly exercise in terms of number of our affected components in our infrastructure)
@bellatrix1001

This comment has been minimized.

Copy link

commented Nov 2, 2018

@dubauski how would you introduce "non-empty passwords"? There doesn't appear to be a way to specify a keystore password in Elastic (but for some reason, that is an option for Logstash).

@dubauski

This comment has been minimized.

Copy link
Author

commented Nov 5, 2018

@bellatrix1001 I was going to try something along the lines of https://www.elastic.co/guide/en/x-pack/current/security-getting-started.html. However, it could be a chicken and the egg problem since you need to be able to start Elastic Search before you can update 'elastic' user's password.

@vnextcoder

This comment has been minimized.

Copy link

commented Mar 16, 2019

Fixed it by installing Oracle JDK and updating the JAVA_HOME in /etc/sysconfig/elasticsearch as per notes in article - https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html .

JAVA_HOME parameters should be updated in the /etc/sysconfig/ only.
also updated SO question- https://stackoverflow.com/questions/30597391/how-to-start-elasticsearch-service-with-multiple-java-jdks/55197805#55197805

@QxiaoQ

This comment has been minimized.

Copy link

commented Oct 10, 2019

I have the same error. Do we have a workaround to use IBM JRE?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.