New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tests] Occasional failures because of "still open connections" in MockTransportService.doClose #22803

Closed
cbuescher opened this Issue Jan 26, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@cbuescher
Member

cbuescher commented Jan 26, 2017

On teardown of the MockTransportService used in unit tests we check that all connections that were opened are closed correctly again. This seems to fail occasionally with errors that look like:

  1> [2017-01-25T22:30:19,038][INFO ][o.e.a.s.RemoteClusterConnectionTests] [testNodeDisconnected]: after test
  2> REPRODUCE WITH: gradle :core:test -Dtests.seed=8A5ADED8D62B415D -Dtests.class=org.elasticsearch.action.search.RemoteClusterConnectionTests -Dtests.method="testNodeDisconnected" -Dtests.security.manager=true -Dtests.locale=ru -Dtests.timezone=America/Tijuana
FAILURE 0.12s J0 | RemoteClusterConnectionTests.testNodeDisconnected <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: still open connections: {{BtOlW7kRTK6YUTDmeeI-cg}{uiX1VgzsQmmqjLTVuqH2HQ}{127.0.0.1}{127.0.0.1:9301}=[org.elasticsearch.test.transport.MockTransportService$5@2cc22be3]}
   > 	at __randomizedtesting.SeedInfo.seed([8A5ADED8D62B415D:430ECAC96D5DA0DF]:0)
   > 	at org.elasticsearch.test.transport.MockTransportService.doClose(MockTransportService.java:763)
   > 	at org.elasticsearch.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:106)
   > 	at org.elasticsearch.action.search.RemoteClusterConnectionTests.testNodeDisconnected(RemoteClusterConnectionTests.java:212)
   > 	at java.lang.Thread.run(Thread.java:745)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/1516/console

Unfortunately I cannot reproduce with the given seed, so just opening this to track this or similar build failures for now.

@s1monw

This comment has been minimized.

Contributor

s1monw commented Jan 26, 2017

I think I know what the issue is. there is a very very small race that is not relevant in production but the tests require more strickness. I will try to improve the code to make sure these failures won't happen anymore

@s1monw s1monw self-assigned this Jan 26, 2017

s1monw added a commit to s1monw/elasticsearch that referenced this issue Jan 26, 2017

Improve connection closing in `RemoteClusterConnection`
Some tests verify that all connection have been closed but due to the
async / concurrent nature of `RemoteClusterConnection` there are situations
where we notify listeners that trigger tests to finish before we actually
closed all connections. The race is very very small and has no impact on the
code correctness. This commit documents and improves the way we close
connections to ensure test won't fail with false positives.

Closes elastic#22803

s1monw added a commit that referenced this issue Jan 26, 2017

Improve connection closing in `RemoteClusterConnection` (#22804)
Some tests verify that all connection have been closed but due to the
async / concurrent nature of `RemoteClusterConnection` there are situations
where we notify listeners that trigger tests to finish before we actually
closed all connections. The race is very very small and has no impact on the
code correctness. This commit documents and improves the way we close
connections to ensure test won't fail with false positives.

Closes #22803

s1monw added a commit that referenced this issue Jan 26, 2017

Improve connection closing in `RemoteClusterConnection` (#22804)
Some tests verify that all connection have been closed but due to the
async / concurrent nature of `RemoteClusterConnection` there are situations
where we notify listeners that trigger tests to finish before we actually
closed all connections. The race is very very small and has no impact on the
code correctness. This commit documents and improves the way we close
connections to ensure test won't fail with false positives.

Closes #22803
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment