Add ASIO completion handler tracking#2681
Merged
Merged
Conversation
SergiySW
previously approved these changes
Jul 16, 2020
SergiySW
self-requested a review
July 16, 2020 07:51
SergiySW
approved these changes
Jul 16, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Boost asio has a way to output diagnostic information about completion handlers by defining
BOOST_ASIO_ENABLE_HANDLER_TRACKINGyou get output like so:Information about what these entries mean can be found here:
https://www.boost.org/doc/libs/1_72_0/doc/html/boost_asio/overview/core/handler_tracking.html
But they can also be turned into a graph: (described later)

By itself it's not that useful as we really care about how long these io thread completion handlers are taking. I've modified the
thread_runnerandsystempolling to now report any completion handlers which take longer thanNANO_ASIO_HANDLER_TRACKING(new CMake variable). It outputs the timestamp which can be cross-referenced with the images/text to find what caused it. It is recommended to use less io_threads if trying to diagnose issues with these as multiple completion handlers cause be executing in parallel.Proposed mode of operation:
1 -
cmake -DNANO_ASIO_HANDLER_TRACKING=10(report anything held for 10ms or greater)2 - Lower
io_threadsin config-node.toml3 - Run suspect code and pipe error output (asio uses that stream):
./core_test --gtest_filter=suspect_test 2> asio_output.txt4 - For any "io_thread held for ??ms" messages, cross-reference with text and remove extraneous text surrounding it (see below)
5 - Generate image
perl handlerviz.pl < asio_output.txt | dot -Tpng > output.png. If theasio_output.txtfile is large this can crash or take a very long time, as well as make the image unreadable.dotbinary is from GraphViz and can be installed for free on all platforms.This can be a good first step in helping diagnose issues and inform of any completion handlers taking too long. A further step may be to wrap all completion handlers and break before exiting but that's a much larger effort with greater.