-
Notifications
You must be signed in to change notification settings - Fork 24.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI] org.elasticsearch.xpack.ml.integration.SetUpgradeModeIT SuitTimeOut Failure #49467
Comments
Pinging @elastic/ml-core (:ml) |
The timeout failure is not a one off I've found another recently: https://gradle-enterprise.elastic.co/s/xjcyfh4xigvvm In both cases the suite times out and jstack shows this stack trace on the client:
The test is passing but there is a problem stopping the datafeed in the cleanup. Later tests are failing because the persistent task associated with the datafeed has not stopped and the stop datafeed transport action is still running.
That is interesting because The test toggles ML upgrade mode which stops the job and datafeed. When upgrade mode is turned back on the datafeed automatically restarts and the test succeeds when the datafeed is assigned a node. Test teardown then stops the datafeed. I think the problem is a race condition between the datafeed re-starting and it being stopped in the teardown, i.e. when the datafeed says it has been assigned to a node it has not started then the stop request hangs on a datafeed that has not started. The logs support this:
|
Possibly fixed by #51302 |
Interestingly from #51285 (comment):
This would explain why no NPE was observed in the triage of this issue, and makes it highly likely that #51302 does fix this. Therefore I'll close this optimistically. Please reopen if it happens again. |
Copying from #51302 (comment) the following explains the reason for the timeout:
|
In
7.5
org.elasticsearch.xpack.ml.integration.SetUpgradeModeIT
timed out on CI just now in run https://gradle-enterprise.elastic.co/s/jp5zidernhxdc/console-log#L3830.The text was updated successfully, but these errors were encountered: