GEODE-8521: detect if a p2p reader thread is hung #5763
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 6 commits
November 18, 2020 11:11
-- made some fields final -- removed unused parameters -- marked some methods visible for testing
bschuchardt
reviewed
Nov 20, 2020
There was a problem hiding this comment.
It would be nice to be able to detect stuck P2P Reader threads that are dispatching messages but I'm concerned about the need to take a millisecond clock reading at the beginning of every message-dispatch.
Have you thought about having the thread be monitored continuously and checking its connectionState to see if it's dispatching? You'd have to add a new DISPATCHING state and have custom code in the monitor for the new P2PReaderExecutorGroup to do this checking.
geode-core/src/main/java/org/apache/geode/internal/monitoring/ThreadsMonitoringImpl.java
Outdated
Show resolved
Hide resolved
This does nothing on the base class but on P2PReaderExecutorGroup it turns monitoring on and off. Also the monitor thread will now detect a zero startTime and in that case set the startTime allowing threads being monitored to not keep setting it themselves (which happens more often).
will now always be the values configured on the DistributedSystem. Previously the thread monitoring classes had their own instance of DistributionConfigImpl which at least in some cases could be inconsistent the the config on the DistributedSystem. Now these two config values are passed in to the constructor. 2. If an instance of ResourceManagerStats can not be found then it will now only prevent the monitor from updating the "numThreadsStuck" stat. Before it prevented it from doing any detection and logging. The lazy initialization of the ResourceManagerStats is now cleaner and does not require the integration test to explicitly call run().
agingade
approved these changes
Nov 23, 2020
bschuchardt
approved these changes
Nov 30, 2020
kirklund
requested changes
Nov 30, 2020
geode-core/src/main/java/org/apache/geode/internal/monitoring/ThreadsMonitoringProcess.java
Outdated
Show resolved
Hide resolved
...re/src/test/java/org/apache/geode/internal/monitoring/ThreadsMonitoringProcessJUnitTest.java
Outdated
Show resolved
Hide resolved
...test/java/org/apache/geode/internal/monitoring/executor/P2PReaderExecutorGroupJUnitTest.java
Outdated
Show resolved
Hide resolved
sometimes due to it creating a background thread that could interfere with unit testing.
kirklund
approved these changes
Nov 30, 2020
|
the dunit failure is the known issue: GEODE-8573 |
mkevo
pushed a commit
to Nordix/geode
that referenced
this pull request
Apr 8, 2021
* AbstractExecutor can now be suspended and resumed. This does nothing on the base class but on P2PReaderExecutorGroup it turns monitoring on and off. Also the monitor thread will now detect a zero startTime and in that case set the startTime allowing threads being monitored to not keep setting it themselves (which happens more often). * the timeInterval and timeLimit that come from gemfire properties will now always be the values configured on the DistributedSystem. Previously the thread monitoring classes had their own instance of DistributionConfigImpl which at least in some cases could be inconsistent the the config on the DistributedSystem. Now these two config values are passed in to the constructor. * If an instance of ResourceManagerStats can not be found then it will now only prevent the monitor from updating the "numThreadsStuck" stat. Before it prevented it from doing any detection and logging. The lazy initialization of the ResourceManagerStats is now cleaner and does not require the integration test to explicitly call run(). Co-authored-by: Darrel Schneider <darrel@vmware.com>
mkevo
pushed a commit
to Nordix/geode
that referenced
this pull request
Apr 19, 2021
* AbstractExecutor can now be suspended and resumed. This does nothing on the base class but on P2PReaderExecutorGroup it turns monitoring on and off. Also the monitor thread will now detect a zero startTime and in that case set the startTime allowing threads being monitored to not keep setting it themselves (which happens more often). * the timeInterval and timeLimit that come from gemfire properties will now always be the values configured on the DistributedSystem. Previously the thread monitoring classes had their own instance of DistributionConfigImpl which at least in some cases could be inconsistent the the config on the DistributedSystem. Now these two config values are passed in to the constructor. * If an instance of ResourceManagerStats can not be found then it will now only prevent the monitor from updating the "numThreadsStuck" stat. Before it prevented it from doing any detection and logging. The lazy initialization of the ResourceManagerStats is now cleaner and does not require the integration test to explicitly call run(). Co-authored-by: Darrel Schneider <darrel@vmware.com>
alb3rtobr
pushed a commit
to Nordix/geode
that referenced
this pull request
Aug 2, 2021
* AbstractExecutor can now be suspended and resumed. This does nothing on the base class but on P2PReaderExecutorGroup it turns monitoring on and off. Also the monitor thread will now detect a zero startTime and in that case set the startTime allowing threads being monitored to not keep setting it themselves (which happens more often). * the timeInterval and timeLimit that come from gemfire properties will now always be the values configured on the DistributedSystem. Previously the thread monitoring classes had their own instance of DistributionConfigImpl which at least in some cases could be inconsistent the the config on the DistributedSystem. Now these two config values are passed in to the constructor. * If an instance of ResourceManagerStats can not be found then it will now only prevent the monitor from updating the "numThreadsStuck" stat. Before it prevented it from doing any detection and logging. The lazy initialization of the ResourceManagerStats is now cleaner and does not require the integration test to explicitly call run(). Co-authored-by: Darrel Schneider <darrel@vmware.com>
alb3rtobr
pushed a commit
to Nordix/geode
that referenced
this pull request
Aug 4, 2021
* AbstractExecutor can now be suspended and resumed. This does nothing on the base class but on P2PReaderExecutorGroup it turns monitoring on and off. Also the monitor thread will now detect a zero startTime and in that case set the startTime allowing threads being monitored to not keep setting it themselves (which happens more often). * the timeInterval and timeLimit that come from gemfire properties will now always be the values configured on the DistributedSystem. Previously the thread monitoring classes had their own instance of DistributionConfigImpl which at least in some cases could be inconsistent the the config on the DistributedSystem. Now these two config values are passed in to the constructor. * If an instance of ResourceManagerStats can not be found then it will now only prevent the monitor from updating the "numThreadsStuck" stat. Before it prevented it from doing any detection and logging. The lazy initialization of the ResourceManagerStats is now cleaner and does not require the integration test to explicitly call run(). Co-authored-by: Darrel Schneider <darrel@vmware.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Each p2p reader thread now creates a single instance of AbstractExecutor
and registers it with the ThreadsMonitoring instance while it is "dispatching"
the message. It will not be monitored while it is waiting on the socket to read
a message. So this will only report stuck p2p readers that are stuck dispatch
which for inline processing includes processing the message and reading the
direct ack.
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
Has your PR been rebased against the latest commit within the target branch (typically
develop)?Is your initial contribution a single, squashed commit?
Does
gradlew buildrun cleanly?Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.