Test failure: org.elasticsearch.grok.GrokTests.testExponentialExpressions

[jakelandis]

Given the rarity of this failure I am not muting. However, if this increases in occurrences this test should be muted.

Build scan:

https://gradle-enterprise.elastic.co/s/iak3abjnc4huu

Repro line:

REPRODUCE WITH: ./gradlew ':libs:elasticsearch-grok:test' --tests "org.elasticsearch.grok.GrokTests.testExponentialExpressions" \
  -Dtests.seed=2012EF738111106E \
  -Dtests.security.manager=true \
  -Dtests.locale=es-VE \
  -Dtests.timezone=Africa/Niamey \
  -Druntime.java=11

Reproduces locally?:
No. Ran a few thousand iterations (without the extra -D arguments ) and they all passed.

Applicable branches:

This failure is on master, but history shows it is not specific to master.

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.relativeStartTime=P28D&search.timeZoneId=America/Chicago&tests.container=org.elasticsearch.grok.GrokTests&tests.sortField=FAILED&tests.test=testExponentialExpressions&tests.unstableOnly=true

This is the only one in past 28 day.

However, searching my email archives i see ~ 15 failures in the past 1.5 years across all branches. So this is pretty rare, but not also not likely caused by cosmic rays. I suspect that there is a subtle race condition in the test itself.

Failure excerpt:

org.elasticsearch.grok.GrokTests > classMethod FAILED	
    java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).	
        at __randomizedtesting.SeedInfo.seed([2012EF738111106E]:0)	

and

2> WARNING: Suite execution timed out: org.elasticsearch.grok.GrokTests 

I believe that for this particular test, it can occur if the Watch dog thread does not kill the Grok match as it should.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[dakrone]

Seeing as how we haven't had more occurrences of this since last July, I'm going to close this for now, we can reopen if needed.

[DaveCTurner]

I had a failure of this in a PR build: https://gradle-enterprise.elastic.co/s/ff62t5dy33ux2/console-log?task=:libs:elasticsearch-grok:test

[fcofdez]

Looks like this is happening again https://gradle-enterprise.elastic.co/s/biozovvlmi3dk

[joegallo]

"Thread-11715" ID=11760 TIMED_WAITING
	at java.base@20.0.1/java.lang.Thread.sleep0(Native Method)
	at java.base@20.0.1/java.lang.Thread.sleep(Thread.java:484)
	at app//org.elasticsearch.grok.GrokTests.lambda$testExponentialExpressions$2(GrokTests.java:776)
	at app//org.elasticsearch.grok.GrokTests$$Lambda$523/0x000000080132fd40.accept(Unknown Source)
	at app//org.elasticsearch.grok.MatcherWatchdog$Default.interruptLongRunningExecutions(MatcherWatchdog.java:142)
	at app//org.elasticsearch.grok.MatcherWatchdog$Default$$Lambda$528/0x000000080133f618.run(Unknown Source)
	at app//org.elasticsearch.grok.GrokTests.lambda$testExponentialExpressions$1(GrokTests.java:783)
	at app//org.elasticsearch.grok.GrokTests$$Lambda$527/0x000000080133f400.run(Unknown Source)
	at java.base@20.0.1/java.lang.Thread.runWith(Thread.java:1636)
	at java.base@20.0.1/java.lang.Thread.run(Thread.java:1623)

It's interesting to me that in the failures I don't see a thread like the above (in the output of the threads that are running when the timeout is reached). That makes me think that for some reason the watchdog didn't execute at all.

edit: Or it executed and died in some interesting way that we're not capturing.

[albertzaharovits]

Here's another case of this https://gradle-enterprise.elastic.co/s/eiooe5kakotua .

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[joegallo]

Still hoping for a new reproduction now that #96230 has been merged. No luck yet.

[mark-vieira]

@joegallo got another one for you: https://gradle-enterprise.elastic.co/s/w6ne6rvsehkpe/tests/:libs:elasticsearch-grok:test/org.elasticsearch.grok.GrokTests/testExponentialExpressions

[joegallo]

Ahhhh! How exciting! Thanks, @mark-vieira!