Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kerberos-tests fails due to ES not starting up: SettingsException: Failed to get setting group for [] setting prefix and setting [type] because of a missing '.' #35942

Closed
alpar-t opened this issue Nov 27, 2018 · 1 comment · Fixed by #35986
Closed

kerberos-tests fails due to ES not starting up: SettingsException: Failed to get setting group for [] setting prefix and setting [type] because of a missing '.' #35942

alpar-t opened this issue Nov 27, 2018 · 1 comment · Fixed by #35986
Assignees
@tvernum
Labels
:Security/Security Security issues without another label >test-failure Triaged test failures from CI v7.0.0-beta1

Comments

@alpar-t
Copy link
Contributor

alpar-t commented Nov 27, 2018

Reproduce with (vagrant support required):

./gradlew x-pack/qa/kerberos-tests check

Fails with:

> Task :x-pack:qa:kerberos-tests:integTestCluster#wait FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':x-pack:qa:kerberos-tests:integTestCluster#wait'.
> java.lang.reflect.UndeclaredThrowableException (no error message)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 3m 30s
262 actionable tasks: 232 executed, 30 up-to-date
[:~alpar/work/elastic/elasticsearch/x-pack/qa/kerberos-tests] master+ 3m31s 1 ± cat build/cluster/integTestCluster\ node0/cwd/run.log
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2018-11-27T12:45:30,193][INFO ][o.e.e.NodeEnvironment    ] [node-0] using [1] data paths, mounts [[/ (/dev/mapper/secureSystem)]], net usable_space [15.4gb], net total_space [223.8gb], types [ext4]
[2018-11-27T12:45:30,195][INFO ][o.e.e.NodeEnvironment    ] [node-0] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-11-27T12:45:30,196][INFO ][o.e.n.Node               ] [node-0] node name [node-0], node ID [am--T0nMSMCGizlsFIpEDQ]
[2018-11-27T12:45:30,196][INFO ][o.e.n.Node               ] [node-0] version[7.0.0-SNAPSHOT], pid[1587], build[default/zip/b6eb73a/2018-11-27T10:43:14.601148Z], OS[Linux/4.18.16-arch1-1-ARCH/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11/11+28]
[2018-11-27T12:45:30,196][INFO ][o.e.n.Node               ] [node-0] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.LsZ04dTS, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=unpooled, -Djava.security.krb5.conf=/home/alpar/work/elastic/elasticsearch/test/fixtures/krb5kdc-fixture/build/conf/krb5.conf, -Dsun.security.krb5.debug=true, -ea, -esa, -Des.path.home=/home/alpar/work/elastic/elasticsearch/x-pack/qa/kerberos-tests/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT, -Des.path.conf=/home/alpar/work/elastic/elasticsearch/x-pack/qa/kerberos-tests/build/cluster/integTestCluster node0/elasticsearch-7.0.0-SNAPSHOT/config, -Des.distribution.flavor=default, -Des.distribution.type=zip]
[2018-11-27T12:45:30,196][WARN ][o.e.n.Node               ] [node-0] version [7.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2018-11-27T12:45:30,538][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:608) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:306) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:306) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	... 6 more
Caused by: org.elasticsearch.common.settings.SettingsException: Failed to get setting group for [] setting prefix and setting [type] because of a missing '.'
	at org.elasticsearch.common.settings.Settings.getGroupsInternal(Settings.java:494) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Settings.getAsGroups(Settings.java:510) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.common.settings.Settings.getAsGroups(Settings.java:506) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$getRealmsSSLSettings$4(SSLService.java:623) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1336) ~[?:?]
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1503) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.getRealmsSSLSettings(SSLService.java:622) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:406) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:97) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:599) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:550) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:465) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:157) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:306) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
	... 6 more

Note that this test will run in CI only if the worker is bare metal which is very unlikely these days.
@alpar-t alpar-t added >test-failure Triaged test failures from CI v7.0.0 :Security/Security Security issues without another label labels Nov 27, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

tvernum added a commit to tvernum/elasticsearch that referenced this issue Nov 28, 2018
@tvernum
Fix kerberos setting registration
c2b977d 
In elastic#30241 Realm settings were changed, but the Kerberos realm settings
were not registered correctly. This change fixes the registration of
those Kerberos settings.

Also adds a new integration test that ensures every internal realm can
be configured in a test cluster.

Also fixes the QA test for kerberos.

Resolves: elastic#35942
@tvernum tvernum mentioned this issue Nov 28, 2018
Merged
tvernum added a commit that referenced this issue Nov 29, 2018
@tvernum
Fix kerberos setting registration (#35986)
609f742 
In #30241 Realm settings were changed, but the Kerberos realm settings
were not registered correctly. This change fixes the registration of
those Kerberos settings.

Also adds a new integration test that ensures every internal realm can
be configured in a test cluster.

Also fixes the QA test for kerberos.

Resolves: #35942
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tvernum tvernum

Labels
:Security/Security Security issues without another label >test-failure Triaged test failures from CI v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix kerberos setting registration tvernum/elasticsearch
4 participants
@colings86 @tvernum @alpar-t @elasticmachine