Timeout for exchange_peer_info and fix for AM tests
#994
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After some debugging of Distributed tests with UCX it was observed that sometimes `exchange_peer_info` hangs indefinitely, specifically when executing `stream_recv` on the client side. The causes for this is unknown but believed to be due to messages being lost if there's either multiple stream messages being transferred simultaneously among various endpoints or being lost due to the receiving end taking too long to launch `stream_recv`, see rapidsai#509 for a similar issue related to stream API. By adding a timeout doesn't allow recovery, but at least allows a UCX-Py client to retry upon failure to establish the endpoint.
|
@charlesbluca could you plan on rebuilding gpuCI's docker image once this is merged in? It should fix dask/distributed#5229 , or it seems to do so at least. |
|
Yup can rebuild once this is reflected in ucx-py nightlies |
This should resolve rapidsai#797, which seems to be caused by checking received messages too early, before they are actually received by the listener. A roundtrip ensures the client receives the reply and thus prevents us from the checking for a transfer that didn't complete yet.
This change should prevent `DeprecationWarning`s from pytest-asyncio which currently closes unclosed event loop but will stop doing that in future releases.
pentschev
changed the title
exchange_peer_infoexchange_peer_info and fix for AM tests
|
@rapidsai/ucxx-python-codeowners could I please get your review here ASAP? I have the expectation that this may improve the situation with Dask-CUDA tests, thus getting it merged soon will let us verify that. |
raydouglass
approved these changes
Oct 11, 2023
charlesbluca
approved these changes
Oct 11, 2023
ci/test_wheel.sh
Outdated
| @@ -14,4 +14,4 @@ cd tests | |||
| python -m pytest --cache-clear -vs . | |||
| cd ../ucp | |||
| # skipped test context: https://github.com/rapidsai/ucx-py/issues/797 | |||
There was a problem hiding this comment.
Should be good to remove this if this closes out #797:
Suggested change
| # skipped test context: https://github.com/rapidsai/ucx-py/issues/797 |
There was a problem hiding this comment.
Thanks @charlesbluca , fixed in 6a8ebf2. It seems that you're not in the ucxpy-python-codeowners group though, was this always the case? I really thought you were in that group.
|
/merge |
pentschev
added a commit
to pentschev/distributed
that referenced
this pull request
Oct 13, 2023
Some time ago `test_ucx_config_w_env_var` started failing intermittently, and the causes were still unknown. After some investigation it seems in certain cases exchanging UCX-Py peer information causes some of the underlying communication calls to never complete and thus cause a hang that can't be recovered from by Distributed. With rapidsai/ucx-py#994, UCX-Py now has a timeout on those calls that allow Distributed to catch and retry establishing the connection, which seems to resolve the problem.
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After some debugging of Distributed tests with UCX it was observed that sometimes
exchange_peer_infohangs indefinitely, specifically when executingstream_recvon the client side. The causes for this is unknown but believed to be due to messages being lost if there's either multiple stream messages being transferred simultaneously among various endpoints or being lost due to the receiving end taking too long to launchstream_recv, see #509 for a similar issue related to stream API. By adding a timeout doesn't allow recovery, but at least allows a UCX-Py client to retry upon failure to establish the endpoint.This change seems to resolve dask/distributed#5229, at least it isn't reproducible locally with this change.
Additionally do a roundtrip message transfer for `test_send_recv_am, which should resolve #797 and seems to be caused by checking received messages too early, before they are actually received by the listener. A roundtrip ensures the client receives the reply and thus prevents us from the checking for a transfer that didn't complete yet.
Ensure now also that the listener is closed before validating
test_close_callbackconditions, which was also flaky.Finally, ensure we close the loop in test fixture, thus preventing
DeprecationWarnings from pytest-asynciowhich currently closes unclosed event loop but will stop doing that in future releases.
Closes #797