Run C++ tests on CI with run_test.py #99956
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99956
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit 4d545dd: NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
huydhn
added
the
ciflow/periodic
huydhn
added
the
ciflow/unstable
| fi | ||
|
|
||
| python test/cpp/jit/tests_setup.py shutdown | ||
| # Cleaning up test artifacts in the test folder | ||
| pushd test |
There was a problem hiding this comment.
I'm curious why the pushd/popd was needed now when it wasn't needed before
There was a problem hiding this comment.
This script test/cpp/jit/tests_setup.py creates and cleans up a dummy JIT output for testing to the current directory. Previously, when the C++ tests was run directly as an executable, there was no need to change the directory. On the other hand, run_test.py changes the working directory to test, thus I need to do the same here to create that dummy output in test folder instead.
| fi | ||
| if [[ -x ./cuda_complex_math_test ]]; then | ||
| ./cuda_complex_math_test | ||
| python test/run_test.py --cpp --verbose -i cpp/cuda_complex_math_test |
There was a problem hiding this comment.
Why make these verbose? Does this turn them into "as verbose as normal python tests" mode, or do cpp tests need to be extra verbose for some reason?
There was a problem hiding this comment.
Just a force of habit I guess as other Python tests all run with --verbose. Now that you point it out, there is no difference in this case from what I see between python test/run_test.py --cpp -i cpp/basic --verbose and python test/run_test.py --cpp -i cpp/basic for example.
|
@pytorchbot merge -f 'ROCm distributed tests are flaky and periodic tests are broken. None are C++ tests' |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
The flaky issue has been fixed by #100909. In addition, retry support for C++ is now available after #99956. So it's safe to move this back from unstable now. Per the discussion with @clee2000, it makes sense to have this as a periodic job given that this one rarely fails. To prevent any gap here, I have created pytorch/test-infra#4144 to add the necessary testing for Vulkan change. Pull Request resolved: #101026 Approved by: https://github.com/kit1980
After #99559, we can now run C++ test with
run_test.py. Although advance features such as--import-slow-testsand--import-disabled-testswon't work for now, there will still be a gain in reliability and performance as C++ can now be retried and run in parallel.This covers all C++ tests in the CI including aten, libtorch, and Vulkan C++ tests across all platforms Linux, Windows, MacOS.
Notes:
CPP_TESTS_DIRcan be set to where the C++ test binaries is located--gtest-filterI need to figure out why conftest.py doesn't work when I invoke pytest directly for C++ test, soFound the cause,--scis not available for C++ tests at the moment. Proper pytest plugin like stepwise works fine though. I'll investigate and fix it in a separate PRconftest.pyis per directory and needs to be in any arbitrary directory that holds C++ testtest_apiandtest_tensorexprtimed out on ASAN, I suspect that ASAN is now used on top of the python executable, which is slower than running native C++ code. IMO, it's ok to run these tests as before on ASAN for now