New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Report JUnit test results for all TVM Python tests #7450
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you prefer to prefix the test case names e.g.
cython-tests.python.unittest.test_arith_canonical_simplify.test_split_index_simplify
?
I think the prefixing is a good idea, to make that obvious. I never got into the internals of JUnit reporting (just used directly to publish on Jenkins, as you're enabling here) so I don't know how complex that change is.
Out of curiosity, are the test scripts already publishing the xml reports?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Prefixing sounds good to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
I had thought I pushed |
@leandron please explicitly approve if you're ok w/ this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, LGTM - thanks @areusch
* Enable JUnit parsing for Python tests * retrigger CI * prefix junit results with FFI type * remove - in junit prefix
* Enable JUnit parsing for Python tests * retrigger CI * prefix junit results with FFI type * remove - in junit prefix
This change enables JUnitXML output from pytest for all TVM python testing. The goal is to enable us to better understand which TVM tests are slow, how a change might impact them, and the runtime variance we see in the TVM CI.
See the example test report produced from the staging run on this PR.
Note that a previous staging run did timeout in the frontend tests, but by a long shot, so I think there's a chance it was unrelated to this PR.
For discussion
>
button on the graph) at both the test level and at the test group level.pytest
can be configured to add a prefix per-run, so we could fix this easily right now. On the other hand, adding the prefix could clutter the test output and make it harder to scan.cython-tests.python.unittest.test_arith_canonical_simplify.test_split_index_simplify
?@tqchen @junrushao1994 @tkonolige @jroesch @zhiics @comaniac @u99127 @manupa-arm @leandron