Resiliency: Throw exception if the JVM will corrupt data. #7580

rmuir · 2014-09-04T00:34:40Z

Detect the worst-offender, known hotspot versions that can
cause index corruption, and fail on startup.

Detect the worst-offender, known hotspot versions that can cause index corruption, and fail on startup.

dadoonet · 2014-09-04T02:51:11Z

src/main/java/org/elasticsearch/bootstrap/Bootstrap.java

@@ -195,11 +198,19 @@ public static void main(String[] args) {
        }

        String stage = "Initialization";
-        try {
+        try {          


Not needed change :)

I think these changes are unavoidable and auto-created by the eclipse configuration in elasticsearch :(

s1monw · 2014-09-04T06:46:22Z

I left two comments - looks good in general

spinscale · 2014-09-04T07:47:00Z

"Please upgrade" is the last message the user gets in the exception, without having any clue where to upgrade to and which other versions are affected (maybe even the latest one is).

Can we point the user somewhere, where he gets help instead of being unspecific here? A documentation page which contains unsafe versions we could link to or something?

dadoonet · 2014-09-04T08:42:54Z

+1 for @spinscale comment. I really like when projects log errors with links to documentation.

rmuir · 2014-09-04T09:32:01Z

Who will make and maintain such a page and ensure it doesnt go 404?

clintongormley · 2014-09-04T09:40:58Z

@rmuir the docs build process checks all links within the docs before pushing any changes live. it also has hooks for custom checks, eg we also check the links in the config.yml file to ensure that they are still correct. We could add a custom hook for this page too.

rmuir · 2014-09-04T09:44:31Z

I'm not really a webpage designer. I think this is blowing up the scope of the issue with unnecessary requirements.

dadoonet · 2014-09-04T09:46:33Z

I think we should think of something somehow generic. Such as what Twitter4J does.
For each error you get, you have an associated Error code which is unique.

The log says:

Relevant discussions can be found on the Internet at:
    http://www.google.co.jp/search?q=93f2523c or
    http://www.google.co.jp/search?q=8b74a4e3
TwitterException{exceptionCode=[93f2523c-8b74a4e3], statusCode=401, message=Invalid or expired token, code=89, retryAfter=-1, rateLimitStatus=null, version=3.0.3}

I think we should think about something similar but in another issue as we somehow hijacked this one :)

clintongormley · 2014-09-04T09:47:06Z

Well, we need to document good versions anyway. It doesn't require any design, just a page of docs. This is versioned along with the code. You tell me what info needs to be on the page and I'll write it.

rmuir · 2014-09-04T09:48:44Z

Crashing with an empty SIGSEGV is better than today. It prevents data corruption. We don't even need an error message at all, a webpage is over the top. Its stupid to block this issue on such things.

clintongormley · 2014-09-04T12:37:46Z

Rather than adding a link to the error message, which may well change in the future, let's just add "Please see the documentation" to the error message. Searching for "java version" or "jvm version" will bring up the docs section added in #7591

nik9000 · 2014-09-04T13:06:11Z

@clintongormley watch out for how personalized google can be - that probably would work for you or me but it might not work for someone who's searched for elasticsearch fewer times. Besides, if you move the page it should get a redirect. Or you could go all out and point to a version on the internet archive or something....

clintongormley · 2014-09-04T13:29:44Z

@nik9000 i meant the search in the ES docs :)

won't work now, but will when #7591 is merged

nik9000 · 2014-09-04T13:43:42Z

@nik9000 i meant the search in the ES docs :)

Ah! Cool. Can you link to a page that searches the Elasticsearch docs? Something like [this](http://www.elasticsearch.org/guide/?s=java version)?

s1monw · 2014-09-05T20:19:17Z

@rmuir I left some comments - I think it's close though!

s1monw · 2014-09-10T07:44:58Z

@rmuir can we move forward here it is marked as a blocker an I'd really like to move forward here

rmuir · 2014-09-10T17:08:19Z

I don't think the change will make it in. Please understand my point of view:

I didn't just take a few minutes and whip up a quick patch here, i tested extensively with all impacted hotspot versions (both Oracle JDK and openjdk/IcedTea), including testing the workaround bypass, and testing that "good versions" are ok, etc. This took me a good bit of time.

This is an exceptionally dangerous piece of functionality. With all the scope creep of this issue ("extra features" and "nice to haves" etc), comes the need to retest. There is no way I would touch this logic in a functional way, such as renaming parameters, without retesting.

So we should decide what is really a priority, and what can be a followup issue.

Currently I am fighting other fires, and I just dont have enough time to do it right, sorry.

s1monw · 2014-09-11T07:15:23Z

moved out to 1.5

s1monw · 2015-03-20T21:49:54Z

@rmuir @rjernst should we revisit this or should I just close it for now?

s1monw · 2015-03-20T22:04:08Z

I really think we should use what we have and expand in other issues. This PR is a good improvement and it's been sitting here for way too long. @rmuir can you rebase this branch and add maybe missing JVMs? all the other requests can be added later.

rjernst · 2015-03-20T22:05:58Z

+1 to get this in as is.

Conflicts: src/main/java/org/elasticsearch/bootstrap/Bootstrap.java

rmuir · 2015-03-21T06:14:27Z

I updated the check since i had to re-test all JVMs anyway.

I refactored this out of bootstrap, made messaging more verbose, added support for compiler workaround flags, and added logic for IBM (because users have hit this on the mailing list).

Example output with different versions:

1.7.0:

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar org.elasticsearch.bootstrap.JVMCheck

Exception in thread "main" java.lang.RuntimeException: Java version: 1.7.0 suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-7070134 which can cause data corruption.
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.
If you absolutely cannot upgrade, please add -XX:-UseLoopPredicate to the JVM_OPTS environment variable.

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar -XX:-UseLoopPredicate org.elasticsearch.bootstrap.JVMCheck

Mar 21, 2015 1:48:58 AM org.elasticsearch.bootstrap.JVMCheck check
WARNING: Workaround flag -XX:-UseLoopPredicate for bug https://bugs.openjdk.java.net/browse/JDK-7070134 found.
This will result in degraded performance!
Upgrading is preferred, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar -Des.bypass.vm.check=true org.elasticsearch.bootstrap.JVMCheck

Mar 21, 2015 1:59:57 AM org.elasticsearch.bootstrap.JVMCheck check
WARNING: bypassing jvm version check for version [1.7.0], this can result in data corruption!

1.7.0_40:

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar org.elasticsearch.bootstrap.JVMCheck

Exception in thread "main" java.lang.RuntimeException: Java version: 1.7.0_40 suffers from critical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause data corruption.
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.
If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JVM_OPTS environment variable.
Upgrading is preferred, this workaround will result in degraded performance.

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar -XX:-UseSuperWord org.elasticsearch.bootstrap.JVMCheck

Mar 21, 2015 1:51:03 AM org.elasticsearch.bootstrap.JVMCheck check
WARNING: Workaround flag -XX:-UseSuperWord for bug https://bugs.openjdk.java.net/browse/JDK-8024830 found.
This will result in degraded performance!
Upgrading is preferred, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar -Des.bypass.vm.check=true org.elasticsearch.bootstrap.JVMCheck

Mar 21, 2015 1:58:52 AM org.elasticsearch.bootstrap.JVMCheck check
WARNING: bypassing jvm version check for version [1.7.0_40], this can result in data corruption!

IBM:

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar org.elasticsearch.bootstrap.JVMCheck

Exception in thread "main" java.lang.RuntimeException: IBM runtimes suffer from several bugs which can cause data corruption.
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html for current recommendations.

$ java -cp elasticsearch-2.0.0-SNAPSHOT.jar:lib/lucene-core-5.1.0-snapshot-1662607.jar -Des.bypass.vm.check=true org.elasticsearch.bootstrap.JVMCheck

Mar 21, 2015 1:57:16 AM org.elasticsearch.bootstrap.JVMCheck check
WARNING: bypassing jvm version check for version [1.7.0], this can result in data corruption!

kimchy · 2015-03-21T06:43:39Z

LGTM

Detect the worst-offenders, all IBM versions and several known hotspot versions that can cause index corruption, and fail on startup. Provide/detect compiler workarounds when they exist, but warn about performance degradation. In all cases the check can be bypassed completely with a safety switch via undocumented system property (es.bypass.vm.check=true) Closes #7580

clintongormley · 2015-04-04T18:12:03Z

w00t!

Detect the worst-offenders, all IBM versions and several known hotspot versions that can cause index corruption, and fail on startup. Provide/detect compiler workarounds when they exist, but warn about performance degradation. In all cases the check can be bypassed completely with a safety switch via undocumented system property (es.bypass.vm.check=true) Closes elastic#7580

[Bootstrap] Throw exception if the JVM will corrupt data.

6224eb2

Detect the worst-offender, known hotspot versions that can cause index corruption, and fail on startup.

rmuir added v1.4.0 labels Sep 4, 2014

dadoonet reviewed Sep 4, 2014
View reviewed changes

clintongormley added the resiliency label Sep 4, 2014

rmuir closed this Sep 4, 2014

clintongormley reopened this Sep 4, 2014

clintongormley mentioned this pull request Sep 4, 2014

Docs: Added recommended Java versions to the docs #7591

Closed

s1monw removed the review label Sep 5, 2014

clintongormley added the blocker label Sep 8, 2014

clintongormley changed the title ~~[Bootstrap] Throw exception if the JVM will corrupt data.~~ Resiliency: Throw exception if the JVM will corrupt data. Sep 8, 2014

clintongormley added the >breaking label Sep 8, 2014

s1monw added v1.5.0 and removed v1.4.0.Beta1 labels Sep 11, 2014

clintongormley removed the blocker label Sep 26, 2014

rjernst mentioned this pull request Dec 12, 2014

Prevent execution on known-buggy JVMs #8936

Closed

drewr force-pushed the master branch from dcc3da0 to 7c20a8a Compare February 20, 2015 16:48

s1monw added v1.6.0 and removed v1.5.0 labels Mar 17, 2015

rmuir added 2 commits March 21, 2015 00:31

Merge branch 'master' into jvm-checker

94ef673

Conflicts: src/main/java/org/elasticsearch/bootstrap/Bootstrap.java

refactor checking logic to separate class and improve messages/logic

a2cb864

rmuir closed this in 6da99b3 Mar 21, 2015

rmuir added v1.5.0 v1.4.5 and removed v1.6.0 labels Mar 21, 2015

clintongormley added the :Core/Infra/Core Core issues without another label label Apr 11, 2015

jasontedor mentioned this pull request Nov 7, 2016

Remove JDK 7 checks from JVMCheck #21381

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resiliency: Throw exception if the JVM will corrupt data. #7580

Resiliency: Throw exception if the JVM will corrupt data. #7580

rmuir commented Sep 4, 2014

dadoonet Sep 4, 2014

rmuir Sep 4, 2014

s1monw commented Sep 4, 2014

spinscale commented Sep 4, 2014

dadoonet commented Sep 4, 2014

rmuir commented Sep 4, 2014

clintongormley commented Sep 4, 2014

rmuir commented Sep 4, 2014

dadoonet commented Sep 4, 2014

clintongormley commented Sep 4, 2014

rmuir commented Sep 4, 2014

clintongormley commented Sep 4, 2014

nik9000 commented Sep 4, 2014

clintongormley commented Sep 4, 2014

nik9000 commented Sep 4, 2014

s1monw commented Sep 5, 2014

s1monw commented Sep 10, 2014

rmuir commented Sep 10, 2014

s1monw commented Sep 11, 2014

s1monw commented Mar 20, 2015

s1monw commented Mar 20, 2015

rjernst commented Mar 20, 2015

rmuir commented Mar 21, 2015

kimchy commented Mar 21, 2015

clintongormley commented Apr 4, 2015

Resiliency: Throw exception if the JVM will corrupt data. #7580

Resiliency: Throw exception if the JVM will corrupt data. #7580

Conversation

rmuir commented Sep 4, 2014

dadoonet Sep 4, 2014

Choose a reason for hiding this comment

rmuir Sep 4, 2014

Choose a reason for hiding this comment

s1monw commented Sep 4, 2014

spinscale commented Sep 4, 2014

dadoonet commented Sep 4, 2014

rmuir commented Sep 4, 2014

clintongormley commented Sep 4, 2014

rmuir commented Sep 4, 2014

dadoonet commented Sep 4, 2014

clintongormley commented Sep 4, 2014

rmuir commented Sep 4, 2014

clintongormley commented Sep 4, 2014

nik9000 commented Sep 4, 2014

clintongormley commented Sep 4, 2014

nik9000 commented Sep 4, 2014

s1monw commented Sep 5, 2014

s1monw commented Sep 10, 2014

rmuir commented Sep 10, 2014

s1monw commented Sep 11, 2014

s1monw commented Mar 20, 2015

s1monw commented Mar 20, 2015

rjernst commented Mar 20, 2015

rmuir commented Mar 21, 2015

kimchy commented Mar 21, 2015

clintongormley commented Apr 4, 2015