Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NPE in ClusterStatsNodesTests#testIngestStats #48684

Closed
DaveCTurner opened this issue Oct 30, 2019 · 3 comments · Fixed by #48704
Closed

NPE in ClusterStatsNodesTests#testIngestStats #48684

DaveCTurner opened this issue Oct 30, 2019 · 3 comments · Fixed by #48704
Assignees
@talevy
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP :Data Management/Stats Statistics tracking and retrieval APIs >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

testIngestStats fails (reproducibly) like this:

/Users/davidturner/jvms/jdk-12.jdk/Contents/Home/bin/java -ea -Dtests.seed=A856F587762BF5AD -Didea.test.cyclic.buffer.size=1048576 -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ IDEA CE.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ IDEA CE.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/davidturner/src/elasticsearch-master/server/build-idea/classes/test:/Users/davidturner/src/elasticsearch-master/server/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/libs/x-content/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/test/framework/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/libs/cli/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/libs/nio/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/libs/core/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/libs/secure-sm/build-idea/classes/main:/Users/davidturner/src/elasticsearch-master/libs/geo/build-idea/classes/main:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/8.4.0-snapshot-e648d601efb/aa74590851b6fcf536976f75448be52f6ca18a4a/lucene-core-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/8.4.0-snapshot-e648d601efb/1cb225781b19e758d216987e363b77fa4b041174/lucene-analyzers-common-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/8.4.0-snapshot-e648d601efb/cbbf849e24ef0cc61312579acf6d6c5b72c99cf5/lucene-backward-codecs-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/8.4.0-snapshot-e648d601efb/1bd113010c183168d79fbc10a6b590fdacc3fa35/lucene-grouping-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/8.4.0-snapshot-e648d601efb/4e44a435e14d12113ca9193182a302677fda155e/lucene-highlighter-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/8.4.0-snapshot-e648d601efb/eb8eacd015ef81ef2055ada357a92c9751308ef1/lucene-join-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/8.4.0-snapshot-e648d601efb/4dc565203bb1eab0222c52215891e207e7032209/lucene-memory-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/8.4.0-snapshot-e648d601efb/ef596e6d2a7ac9c7dfc6196dad75dc719c81ce85/lucene-misc-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/8.4.0-snapshot-e648d601efb/b0c963e68dd71444f09336258c8f63425514426a/lucene-queries-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/8.4.0-snapshot-e648d601efb/bfab3e9b0467662a8ff969da215dc4a999b73076/lucene-queryparser-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/8.4.0-snapshot-e648d601efb/dadfc90e4cd032f8a4db5cc1e0bdddecea635edb/lucene-sandbox-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/8.4.0-snapshot-e648d601efb/e72dd79d30781e4d05bc8397ae61d0b51d7ad522/lucene-spatial-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/8.4.0-snapshot-e648d601efb/e6b6dbd0526287f25d98d7fe354d5e290c875b8a/lucene-spatial-extras-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/8.4.0-snapshot-e648d601efb/6351edfc6dde2aefd8f6d8ef33ae5a6e08f88321/lucene-spatial3d-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/8.4.0-snapshot-e648d601efb/921dd4ab493b9d70a0b1bf7b0fe8a6790b7e8036/lucene-suggest-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.8.1/ffc7ba8f289428b9508ab484b8001dea944ae603/hppc-0.8.1.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.10.3/2e5366cf1f77ca3bafffecf6e87d30e1d504e959/joda-time-2.10.3.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.2/2ab94758b0276a8a26102adf8d528cf6d0567b9a/t-digest-3.2.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.7/faa8ba85d503da4ab872d17ba8c00da0098ab2f2/spatial4j-0.7.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.locationtech.jts/jts-core/1.15.0/705981b7e25d05a76a3654e597dab6ba423eb79e/jts-core-1.15.0.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.elasticsearch/jna/4.5.1/da10908ae23dc59b19dc258e63aea1c44621dc3a/jna-4.5.1.jar:/Users/davidturner/src/elasticsearch-master/libs/plugin-classloader/build-idea/classes/main:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.17/7a27ea250c5130b2922b86dea63cbb1cc10a660c/snakeyaml-1.17.jar:/Users/davidturner/src/elasticsearch-master/client/sniffer/build-idea/classes/main:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.11/876ead1db19f0c9e79c9789273a3ef8c6fd6c29b/jackson-core-2.8.11.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.11/d9d1c49c5d9d5e46e2aee55f3cdd119286fe0fc1/jackson-dataformat-smile-2.8.11.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.11/2e77c6ff7342cd61ab1ae7cb14ed16aebfc8a72a/jackson-dataformat-yaml-2.8.11.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.11/8b9826e16c3366764bfb7ad7362554f0471046c3/jackson-dataformat-cbor-2.8.11.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar:/Users/davidturner/src/elasticsearch-master/client/rest/build-idea/classes/main:/Users/davidturner/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.7.1/e917f4983144c3b969eb7d3648338ecde5e3ba89/randomizedtesting-runner-2.7.1.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest/2.1/9420ba32c29217b54eebd26ff7f9234d31c3fbb2/hamcrest-2.1.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-test-framework/8.4.0-snapshot-e648d601efb/683a2a32be913b1b8934f533fe9ea6b96a632128/lucene-test-framework-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-codecs/8.4.0-snapshot-e648d601efb/f8da5001e9469d3cbf1358da222104934d069e4/lucene-codecs-8.4.0-snapshot-e648d601efb.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.11/3acb4705652e16236558f0f4f2192cc33c3bd189/commons-codec-1.11.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.elasticsearch/mocksocket/1.2/190ad3f42fd6ab7367bae0c0e14d1dc4e4a3c361/mocksocket-1.2.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.10/7ca2e4276f4ef95e4db725a8cd4a1d1e7585b9e5/httpclient-4.5.10.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.12/21ebaf6d532bc350ba95bd81938fa5f0e511c132/httpcore-4.4.12.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpasyncclient/4.1.4/f3a3240681faae3fa46b573a4c7e50cec9db0d86/httpasyncclient-4.1.4.jar:/Users/davidturner/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore-nio/4.4.12/84cd29eca842f31db02987cfedea245af020198b/httpcore-nio-4.4.12.jar" com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodesTests "Until Failure"
[2019-10-30T12:14:26,720][INFO ][o.e.a.a.c.s.ClusterStatsNodesTests] [testIngestStats] before test
[2019-10-30T12:14:26,982][INFO ][o.e.a.a.c.s.ClusterStatsNodesTests] [testIngestStats] after test
REPRODUCE WITH: ./gradlew 'null' --tests "org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodesTests.testIngestStats" -Dtests.seed=A856F587762BF5AD -Dtests.locale=it-CH -Dtests.timezone=Atlantic/Faeroe

java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([A856F587762BF5AD:30827FE2C7B63C91]:0)
	at org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodesTests.testIngestStats(ClusterStatsNodesTests.java:74)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:835)

[2019-10-30T12:14:27,009][INFO ][o.e.a.a.c.s.ClusterStatsNodesTests] [testNetworkTypesToXContent] before test
[2019-10-30T12:14:27,044][INFO ][o.e.a.a.c.s.ClusterStatsNodesTests] [testNetworkTypesToXContent] after test
NOTE: leaving temporary files on disk at: /private/var/folders/3f/0f8prhv16qx7l9vcpsx7d0y00000gn/T/org.elasticsearch.action.admin.cluster.stats.ClusterStatsNodesTests_A856F587762BF5AD-004
NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1499, maxMBSortInHeap=7.635152858341031, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2dc8c555), locale=it-CH, timezone=Atlantic/Faeroe
NOTE: Mac OS X 10.14.6 x86_64/Oracle Corporation 12 (64-bit)/cpus=8,threads=1,free=222643776,total=270532608
NOTE: All tests run in this JVM: [ClusterStatsNodesTests]

Process finished with exit code 255

Looks like the reason is that createNodeStats() will rarely return null for the ingest stats:

elasticsearch/server/src/test/java/org/elasticsearch/action/admin/cluster/node/stats/NodeStatsTests.java

Lines 442 to 465 in 523cb23

IngestStats ingestStats = null;
if (frequently()) {
IngestStats.Stats totalStats = new IngestStats.Stats(randomNonNegativeLong(), randomNonNegativeLong(), randomNonNegativeLong(),
randomNonNegativeLong());
int numPipelines = randomIntBetween(0, 10);
int numProcessors = randomIntBetween(0, 10);
List<IngestStats.PipelineStat> ingestPipelineStats = new ArrayList<>(numPipelines);
Map<String, List<IngestStats.ProcessorStat>> ingestProcessorStats = new HashMap<>(numPipelines);
for (int i = 0; i < numPipelines; i++) {
String pipelineId = randomAlphaOfLengthBetween(3, 10);
ingestPipelineStats.add(new IngestStats.PipelineStat(pipelineId, new IngestStats.Stats
(randomNonNegativeLong(), randomNonNegativeLong(), randomNonNegativeLong(), randomNonNegativeLong())));
List<IngestStats.ProcessorStat> processorPerPipeline = new ArrayList<>(numProcessors);
for (int j =0; j < numProcessors;j++) {
IngestStats.Stats processorStats = new IngestStats.Stats
(randomNonNegativeLong(), randomNonNegativeLong(), randomNonNegativeLong(), randomNonNegativeLong());
processorPerPipeline.add(new IngestStats.ProcessorStat(randomAlphaOfLengthBetween(3, 10),
randomAlphaOfLengthBetween(3, 10), processorStats));
}
ingestProcessorStats.put(pipelineId,processorPerPipeline);
}
ingestStats = new IngestStats(totalStats, ingestPipelineStats, ingestProcessorStats);
}

I will mute this.

Relates #48485
@DaveCTurner DaveCTurner added :Data Management/Stats Statistics tracking and retrieval APIs >test-failure Triaged test failures from CI :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Oct 30, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

DaveCTurner added a commit that referenced this issue Oct 30, 2019
@DaveCTurner
Mute ClusterStatsNodesTests#testIngestStats (re. #48684)
63831fa
@DaveCTurner
Copy link
Contributor Author

Muted in 63831fa.

@talevy
Copy link
Contributor

talevy commented Oct 30, 2019

thank you David! I will address

talevy added a commit to talevy/elasticsearch that referenced this issue Oct 30, 2019
@talevy
Fix and re-enable ClusterStatsNodesTests#testIngestStats
96c8b66 
This commit creates random NodesStats until IngestStats is not null, so
that it can be properly tested. This is done as to not change the existing
behavior of createNodesStats since it depends on IngestStats being null
at times for serialization testing purposes. I think it is OK to do this since
it is unlikely that it will be null.

relates elastic#48485

closes elastic#48684
@talevy talevy mentioned this issue Oct 30, 2019
Merged
talevy added a commit that referenced this issue Oct 31, 2019
@talevy
Fix and re-enable ClusterStatsNodesTests#testIngestStats (#48704)
23a4e4a 
This commit creates random NodesStats until IngestStats is not null, so
that it can be properly tested. This is done as to not change the existing
behavior of createNodesStats since it depends on IngestStats being null
at times for serialization testing purposes. I think it is OK to do this since
it is unlikely that it will be null.

relates #48485

closes #48684
debadair pushed a commit to debadair/elasticsearch that referenced this issue Nov 13, 2019
@talevy @debadair
Fix and re-enable ClusterStatsNodesTests#testIngestStats (elastic#48704)
1291a6d 
This commit creates random NodesStats until IngestStats is not null, so
that it can be properly tested. This is done as to not change the existing
behavior of createNodesStats since it depends on IngestStats being null
at times for serialization testing purposes. I think it is OK to do this since
it is unlikely that it will be null.

relates elastic#48485

closes elastic#48684
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@talevy talevy

Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP :Data Management/Stats Statistics tracking and retrieval APIs >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix and re-enable ClusterStatsNodesTests#testIngestStats talevy/elasticsearch
3 participants
@talevy @DaveCTurner @elasticmachine