traceback filtering hook #283
Comments
pytestbot
added
the
type: enhancement
make test suite more tolerable against xdist causing warnings itself
pfctdayelise
added
the
topic: tracebacks
|
@RonnyPfannschmidt, |
|
@nicoddemus no, since Foreign frameworks do not use it |
|
HMmm I see, thanks. Could you post a more detailed example, including how the hook would work? |
|
Think of sqlalchemy, django, pyramid When extending those, they tend to add a lot of extra stack that's usually irrelevant, I'm on tablet ATM so I avoid code writeup, however I don't get have any nice idea for the api To me its a post copy py.code item |
|
Is this still something which needs to be implemented in pytest? |
|
yup |
|
To get the discussion rolling, how about this signature for a hook: @pytest.hookspec(firstresult=True)
def pytest_filter_traceback_entry(filename, lineno, locals):
"""When pytest generates a traceback, this hook will be called for each traceback entry. If any
hook returns `True` for any entry, that entry and all entries below it will be hidden from the final
traceback that will be displayed to the users, except if `-vv` is given on the command-line.
"""This hook receives |
|
@nicoddemus I have a custom pytest plugin to run mamba tests. In the spec tests, I also use the expects library. Below you see the output with a forced error in my test suite, using the For this example, could you tell me how many times the callback will be triggered and which arguments I will get for the above? What I would like to see left over from the traceback above after filtering: So everything from pytest itself filtered and everything from the expects library filtered. |
|
@nicoddemus such a hook will not allow to fold framework bits |
I thought this would do the trick (simplified of course): def pytest_filter_traceback_entry(filename, lineno, locals):
import sqlalchemy
return filename.startswith(os.path.dirname(sqlalquemy.__file__))Or even exclude everything from inside def pytest_filter_traceback_entry(filename, lineno, locals):
import site
for path in site.getsitepackages():
if filename.startswith(path):
return True |
|
@nicoddemus something displaying the trace-backs will not know that a framework section is being filtered, we would also be unable to show it again if needed |
|
@nicoddemus i understand why just throwing them away in our pretty limited painful terminal print only ux makes sense, but anything more interactive will just be really bad if its crippled by the limits of our current bad ux |
Not sure I understand why those limitations are a given based on the hook proposed, after all the code which uses the hook itself might hide the tracebacks and keep them around for other purposes if needed.
As I said, we might not just throw them away, but might set an attribute "hidden" or something like that that each UX can deal with appropriately. Another idea: it might make sense to return a category instead, for example "third-party" or "builtin". This way a richer UI (like It seems like you have some requirements, but they are only in your head. It suggest you write down those requirements or at least some vague ideas to help the discussion. |
|
I had the same problem and hope my solution can serve to show an implementation approach or at least a workaround. @pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_makereport(item, call):
outcome = yield
report = outcome.get_result()
if report.when != 'call':
return
# remove noise (stacktrace) from the tenacity module if there is an exception during waiting
try:
reprentries = [x for x in report.longrepr.chain[0][0].reprentries[0:-1] if
"tenacity/__init__.py" not in x.reprfileloc.path and
"concurrent/futures/_base.py" not in x.reprfileloc.path and
"attempt in Wait" not in x.lines[-1]
]
reprentries.append(report.longrepr.chain[0][0].reprentries[-1]) # make sure to keep the last element of the chain
report.longrepr.chain[0][0].reprentries = reprentries
except AttributeError:
pass |
|
I'm working with tensorflow, so without a good way to do this, I get like 19 pages of stuff I really don't care about and only a couple lines of things I do. I really want that long format output with the local variables and stuff. I just couldn't care less about the 19 billion lines of tensorflow's inner workings when I'm really just having a minor dimensionality problem... I did something similar to @MichaelKorn I think, although, I arguably did it in the wrong place based on a stackoverflow comment. It'd be nice if there was a clean interface for this. I definitely understand the hesitation wrt to cutting out parts of the traceback. It's true there's no generalized way to show what developers actually want to see, but giving the developers some way to tell pytest what they want to see seems like a no brainer. |
|
I think a whitelist strategy would be cheap and flexible enough. |
That's a great idea. 👍 We could have a new option for that, which is backwards compatible and probably simple to implement too. |
|
For reference, werkzeug's debugger differentiates library code from main/app code by checking That maybe could be used here too, maybe as a default. User-configurable seems like a good idea too. |
|
This could potentially be a good solution for reducing traceback noise in It would be nice to have a way for filtering tracebacks based on arbitrary rules to make these less noisy, perhaps even a simple list of regex patterns to determine if a function should be removed from the stacktrace. For example, when using beartype, you'll see the function is shown twice, once for the beartype decorator and once for the function method its wrapping. Although this isn't an upstream issue, I'll shamelessly /cc @leycec (beartype maintainer) for his exceptional knowledge (pun intended) on the internals of Python, as he may have already come across this with other exception handling libraries. I'm not familiar enough with the internals of pytest to say how difficult this would be, but if a core dev could chime in with some hints that'd be super helpful. /cc @nicoddemus ? <@beartype(XXX.Loader._coerce_complex_object) at 0x1e5d5420160>:30: in _coerce_complex_object
???
__beartype_args_len = 2
__beartype_func = <function PSCliXmlLoader._coerce_complex_object at 0x000001E5D5420160>
__beartype_get_violation = <function get_beartype_violation at 0x000001E5AE567910>
__beartype_object_140715471122928 = <class 'xml.etree.ElementTree.Element'>
__beartype_pith_0 = <Element 'Obj' at 0x000001E5D53EF1A0>
args = (<XXX.Loader object at 0x000001E5D540D720>, <Element 'Obj' at 0x000001E5D53EF1A0>)
kwargs = {}
XXX\XXX.py:92: in _coerce_complex_object
tn_element = xml_find_exactly_one(element, './TN|./TNRef')
child_nodes = {'DCT': [<Element 'DCT' at 0x000001E5D53EF920>], 'TN': [<Element 'TN' at 0x000001E5D53EF150>]}
element = <Element 'Obj' at 0x000001E5D53EF1A0>
self = <XXX.Loader object at 0x000001E5D540D720>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
element = <Element 'Obj' at 0x000001E5D53EF1A0>, xpath = './TN|./TNRef'
def xml_find_exactly_one(element: ET.Element, xpath: str):
"""."""
found_elements = element.findall(xpath)
> assert len(found_elements), f'expected 1 object, got {len(found_elements)}'
E AssertionError: expected 1 object, got 0
element = <Element 'Obj' at 0x000001E5D53EF1A0>
found_elements = []
xpath = './TN|./TNRef'
XXX\XXX.py:25: AssertionError |
|
@foxx: ...heh. No one will believe me, but @beartype's current traceback pollution is an improvement over the prior behaviour. We currently only add a single frame of pollution; we used to add about twenty billion frames of pollution, just like TensorFlow and company. Users openly wept back then.
That said, doing so incurs a negligible performance penalty. We usually don't care about that, because this is Python. But we care about that in @beartype, because... @beartype. We really do measure and care about overhead denominated in microseconds. To reduce overhead, @beartype should avoid generating
Theoretically, everybody else could do that too. If upstream packages can reliably detect when |
|
@leycec as the code also look at the globals in pytest/src/_pytest/_code/code.py Line 287 in 69f2855 beartype can probably just add the tracebackhide property to the globals of the check functions (presuming they go to a different module) pytest currently doesn't provide a api to verify it is running (intentionally as people break the world with this all the time) |
You... you're right! @beartype actually generates type-checking wrapper functions dynamically in-memory at runtime, which is how we hit ludicrous speeds. But I'm sure we can just add a fake @RonnyPfannschmidt with the clever catch. I'm impressed and disappointed I didn't think of that. 😓 |
|
@leycec its not directly obvious from the code, its code i'm only familiar with since i want it gone since more than a decade now ;P |
|
For what it's worth, pytest does set a |
Originally reported by: Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt)
we need a way to filter the bits of a traceback that are framework code
since in normal development they seem to distract from the real problem
i propose a hook+ soma pis to be able to replace sets of traceback entries by synthetic frames that will just output as note that framework frames have been skipped
and of course, the --fulltrace option would opt out of it
The text was updated successfully, but these errors were encountered: