FusionResizePad6_CUDA fail with OOM error

[zasdfgbnm]

Everytime when I run the entire test suite, I am seeing the following error:

[ RUN      ] NVFuserTest.FusionResizePad6_CUDA
unknown file: Failure
C++ exception with description "CUDA driver error: out of memory
Exception raised from runFusion at /home/gaoxiang/Fuser/csrc/executor.cpp:1358 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x5c (0x7ff1e37bcacc in /home/gaoxiang/pytorch-viable/build/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x64 (0x7ff1e3785644 in /home/gaoxiang/pytorch-viable/build/lib/libc10.so)
frame #2: nvfuser::FusionExecutor::runFusion(nvfuser::KernelArgumentHolder&, nvfuser::LaunchParams const&, nvfuser::CompileParams, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) + 0x39ac (0x7ff217be919c in /home/gaoxiang/pytorch-viable/build/lib/libnvfuser_codegen.so)
frame #3: <unknown function> + 0x28fa9b (0x55a8fe871a9b in ./build/bin/nvfuser_tests)
frame #4: <unknown function> + 0x28fbf0 (0x55a8fe871bf0 in ./build/bin/nvfuser_tests)
frame #5: <unknown function> + 0x4a2f90 (0x55a8fea84f90 in ./build/bin/nvfuser_tests)
frame #6: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x87 (0x55a8feba2ea7 in ./build/bin/nvfuser_tests)
frame #7: <unknown function> + 0x5b5826 (0x55a8feb97826 in ./build/bin/nvfuser_tests)
frame #8: <unknown function> + 0x5b5a62 (0x55a8feb97a62 in ./build/bin/nvfuser_tests)
frame #9: <unknown function> + 0x5b615a (0x55a8feb9815a in ./build/bin/nvfuser_tests)
frame #10: testing::internal::UnitTestImpl::RunAllTests() + 0x6d4 (0x55a8feb98b14 in ./build/bin/nvfuser_tests)
frame #11: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) + 0x87 (0x55a8feba3417 in ./build/bin/nvfuser_tests)
frame #12: testing::UnitTest::Run() + 0x91 (0x55a8feb97b81 in ./build/bin/nvfuser_tests)
frame #13: <unknown function> + 0x1b8572 (0x55a8fe79a572 in ./build/bin/nvfuser_tests)
frame #14: <unknown function> + 0x23790 (0x7ff1e2e3c790 in /usr/lib/libc.so.6)
frame #15: __libc_start_main + 0x8a (0x7ff1e2e3c84a in /usr/lib/libc.so.6)
frame #16: _start + 0x25 (0x55a8fe7c8375 in ./build/bin/nvfuser_tests)
" thrown in the test body.
[  FAILED  ] NVFuserTest.FusionResizePad6_CUDA (15925 ms)

This failure is pretty reproducible; it fails on my system almost always. I checked FusionResizePad6_CUDA and didn't see anything that uses large memory, so I think FusionResizePad6_CUDA is not to the test blame.

[naoyam]

I'll take a look.

[zasdfgbnm]

I think adding c10::cuda::CUDACachingAllocator::emptyCache() to tests that have heavy GPU memory usage like FusionCompileIndexType_CUDA might help, testing.

[naoyam]

Closed by #49