Add support for sys.monitoring events.
#9482
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Python 3.12 introduced a new event monitoring system `sys.monitoring`. This patch augments Numba's dispatcher so as to emit events associated with a Python function starting and returning from execution. Tools monitoring for such events will therefore be able to identify execution of Numba's pure machine code regions as though they were actual Python functions. This allows tools such a ``cProfile`` to record Numba compiled function execution and report them as part of their output. Fixes numba#9289
sklam
reviewed
Mar 8, 2024
|
nevermind, it's just typical "leak" in original messageValgrind showing these when running |
sklam
reviewed
Apr 3, 2024
Adds support to Numba's dispatcher for the ``sys.monitoring.events`` of type ``RAISE`` and `PY_UNWIND``. Associated unit tests are added and cProfile testing is updated. Docs are updated to match.
Resolved conflicts: numba/tests/support.py
If a Numba function is being run under cProfile and the function raises an exception, the dispatcher must handle this correctly. It is not valid to call `PyFrame_FastToLocals` without saving and restoring the exception state across the call as it can clear the exception state itself. This patch fixes this problem and also stops calling `PyFrame_LocalsToFast` in Python 3.11 where the "whats-new" docs claim that the frames are now looked after by the virutal machine.
stuartarchibald
added
3 - Ready for Review
Effort - long
Fixes some RST syntax and a couple of grammatical errors.
41 tasks
|
BFID |
sklam
added
the
Pending BuildFarm
|
There's an elusive bug that is only revealed by minimal test sequence: python runtests.py numba.tests.test_sys_monitoring.TestMonitoring.test_disable_from_callback numba.tests.test_sys_monitoring.TestMonitoring.test_start_eventoutput: |
sklam
reviewed
Apr 24, 2024
There was a problem hiding this comment.
#9482 (comment) is the only pending problem and I have no idea why it is happening.
Code and documentation looks great
I can reproduce this locally too. |
This works around a potential bug in CPython where the state associated with a monitoring tool can "leak" in an unexpected manner. This patch also removes the use of the functools cache on the generator for test functions, this to eliminate any potential state being stored on the code objects associated with the test functions.
stuartarchibald
added
4 - Waiting on reviewer
|
Note for the future. Were this feature request implemented python/cpython#111997 the code in this patch could be adapted to use it opposed to having to rely on CPython internal details. Edit: This PR python/cpython#116413 implements the above. Looks like it will be in Python 3.13. |
sklam
reviewed
Apr 26, 2024
numba/tests/test_sys_monitoring.py
Outdated
| # It is necessary to restart events that have been disabled. The | ||
| # "disabled" state of the `PY_START` event for the tool | ||
| # `self.tool_id` "leaks" into subsequent tests. These tests then end | ||
| # up failing as events that should trigger do not! It's not really |
There was a problem hiding this comment.
# up failing as events that should trigger do not! It's not really
^^^^^^^
you mean that should not trigger?
There was a problem hiding this comment.
The comment is trying to convey that "subsequent tests fail as events in them that should be triggered are not being triggered" (this was why the Mock.call_count value was 1 too small, an event that should have triggered did not).
There was a problem hiding this comment.
My english parser segfaulted when reading it.
There was a problem hiding this comment.
I've tried to make this comment more clear in 25535b3. Please take a look and see what you think. Thanks!
|
Reported issue to python/cpython#118327 |
As title.
Thanks for reporting this @sklam, much appreciated. |
sklam
approved these changes
Apr 26, 2024
sklam
added
5 - Ready to merge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Python 3.12 introduced a new event monitoring system
sys.monitoring. This patch augments Numba's dispatcher so as to emit events associated with a Python function starting and returning from execution. Tools monitoring for such events will therefore be able to identify execution of Numba's pure machine code regions as though they were actual Python functions. This allows tools such acProfileto record Numba compiled function execution and report them as part of their output.Fixes #9289