Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure SLM stats does not block an in-place upgrade from 7.4 #48367

Merged
merged 2 commits into from
Oct 23, 2019

Conversation

jakelandis
Copy link
Contributor

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]


Note - this is was not caught by normal full cluster state restart tests since by default there is no SLM data in cluster state.

Below is the full error when the test is run without the fix:

./gradlew :x-pack:qa:full-cluster-restart:v7.4.0#bwc -Dtests.class=org.elasticsearch.xpack.restart.FullClusterRestartIT -Dtests.method="testSlmStats"

> Task :printGlobalBuildInfo UP-TO-DATE
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 5.6.2
  OS Info               : Mac OS X 10.14.6 (x86_64)
  JDK Version           : 12 (Oracle Corporation 12.0.1 [Java HotSpot(TM) 64-Bit Server VM 12.0.1+12])
  JAVA_HOME             : /Users/jakelandis/workspace/java/jdk-12.0.1.jdk/Contents/Home
  Random Testing Seed   : 2B2BAE156D0301A9
=======================================

> Task :x-pack:plugin:core:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :x-pack:qa:full-cluster-restart:compileTestJava
Note: /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/src/test/java/org/elasticsearch/xpack/restart/MlMigrationFullClusterRestartIT.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :x-pack:qa:full-cluster-restart:v7.4.0#upgradedClusterTest

=== Standard output of node `node{:x-pack:qa:full-cluster-restart:v7.4.0-0}` ===

»    ↓ errors and warnings from /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/logs/es.stdout.log ↓
» WARN ][o.e.d.FileBasedSeedHostsProvider] [v7.4.0-0] expected, but did not find, a dynamic hosts list at [/Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/config/unicast_hosts.txt]
»   ↑ repeated 4 times ↑
» ERROR][o.e.g.GatewayMetaState   ] [v7.4.0-0] failed to read or upgrade local state, exiting...
»  org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 13 more
»  Caused by: java.lang.IllegalArgumentException: Required [stats]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.finish(ConstructingObjectParser.java:446) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.xpack.ilm.IndexLifecycle.lambda$getNamedXContent$3(IndexLifecycle.java:205) ~[?:?]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry$Entry.lambda$new$0(NamedXContentRegistry.java:63) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1326) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1371) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1362) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:302) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 13 more
» WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.4.0-0] uncaught exception in thread [main]
»  org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:138) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 6 more
»  Caused by: java.io.IOException: failed to read /Users/jakelandis/workspace/7x/elasticsearch/x-pack/qa/full-cluster-restart/build/testclusters/v7.4.0-0/data/_state/global-22.st
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 6 more
»  Caused by: java.lang.IllegalArgumentException: Required [stats]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.finish(ConstructingObjectParser.java:446) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:169) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.xpack.ilm.IndexLifecycle.lambda$getNamedXContent$3(IndexLifecycle.java:205) ~[?:?]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry$Entry.lambda$new$0(NamedXContentRegistry.java:63) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:141) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.common.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:385) ~[elasticsearch-x-content-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1326) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1371) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1362) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:302) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:80) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:89) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.node.Node.start(Node.java:690) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:271) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:348) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»       ... 6 more

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

Copy link
Contributor

@gwbrown gwbrown left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once CI is happy, thanks again for taking care of this. We'll likely want to expand on the restart test at some point, but that can be done in another PR.

@jakelandis
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/1
@elasticmachine run elasticsearch-ci/2
@elasticmachine run elasticsearch-ci/bwc
@elasticmachine run elasticsearch-ci/default-distro
@elasticmachine run elasticsearch-ci/packaging-sample-matrix

@jakelandis
Copy link
Contributor Author

@elasticmachine update branch

Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for fixing this Jake!

@jakelandis jakelandis merged commit f691daa into elastic:master Oct 23, 2019
@jakelandis jakelandis deleted the slm_stats_fix_master branch October 23, 2019 15:24
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Oct 23, 2019
…#48367)

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
jakelandis added a commit to jakelandis/elasticsearch that referenced this pull request Oct 23, 2019
…#48367)

7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@jakelandis
Copy link
Contributor Author

backported via #48411 and #48412

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants