Deconstruction of ThreadLocalStore<T> when program exits leads program crashed

[mz24cn]

Environment info

Operating System:
Windows 10
GPU is GTX850M, CUDA version: 8.0

Compiler:
VS 2015

Package used (Python/R/Scala/Julia):
MXNet.cpp

MXNet version:
Or if installed from source:
yes

Error Message:

The program throws exception in main().

Minimum reproducible example

following 4 lines C++ code makes program crash:
#include <cuda_runtime.h>
#include <curand.h>
#include "mxnet-cpp/MxNetCpp.h"
int main(int argc, char** argv) {
Context ctx_dev(DeviceType::kGPU, 0);
NDArray rand(Shape(512, 28), ctx_dev, false); //this 'NDArray' is MXNet.cpp NDArray
NDArray::SampleGaussian(0, 1, &rand);
NDArray::WaitAll();
}

What have you tried to solve it?

1, NDArray::SampleGaussian use cudaMallocPitch() apply GPU memory;
2, When 4 lines code executed, the program 'prepared' to exit, some global static objects were going to be released;
3, CUDA runtime is also unloading since the program prepared to exit;
4, dmlc::ThreadLocalStoremxnet::resource::ResourceManagerImpl global static object was released. (thread_local.h, line 57);
5, The registered pointers were released (thread_local.h, line 52);
6, Eventually cudaFree() was called to release the memory applied in step 1 (tensor_gpu-int.h, line 54);
7, cudaErrorCudartUnloading was returned by cudaFree() since the reason of step 3;
8, dmlc::Error was thrown (base.h, line 219);
9, Windows reports the program crashed.

dmlc::ThreadLocalStoremxnet::resource::ResourceManagerImpl deconstruction was called by OS, and the sequence of unloading mxnet variables and CUDA runtime cannot be controlled. Maybe some design should be changed a little.

[mz24cn]

Remove last line NDArray::WaitAll(); may avoid of the exception, because cudaFree() was called before CUDA unloaded. Or substitute last line to be 'std::this_thread::sleep_for(std::chrono::milliseconds(1000));', the exception occurs, too.

[mz24cn]

I solved the issue. MXNotifyShutdown() should be called when all computation finished.