improved assert message in the case of "CUDA error: device-side assert triggered"

[stas00]

When pytorch spits out CUDA error: device-side assert triggered would it be possible to add a brief note, such as: "It's not possible to recover. Program/kernel restart is required to continue"

So that users won't waste time trying to recover or re-run their code, when we know ahead of time they won't be able to.

And a bonus feature is to add a recommendation to re-run with CUDA_LAUNCH_BLOCKING=1 env var for debugging the issue.

Thank you.

I have just added in my code (in addition to the normal exception):

                if "device-side assert triggered" in str(e):
                    warn("""When 'device-side assert triggered' error happens, it's not possible to recover and you must restart the kernel to continue. Use os.environ['CUDA_LAUNCH_BLOCKING']="1" before restarting to debug""")

but ideally it should be part of the pytorch exception in first place.

cc @ngimel

[ezyang]

Seems reasonable to me. You'd have to edit C10_CUDA_CHECK, I believe.