Unittest of cudnnCTCLoss failed in cuda10

[wanghaoshuang]

The unittest of warp-CTC op will fail if use_cudnn was true with cuda10. It seems that the gradients given by cudnnCTCloss is different between cuda8 and cuda10.

The cudnn kernel of warpctc has been removed in #19730. And the implement of baidu/research is enough for applications in speech or NLP fields.

But it is also necessary to check the reason of failure in cuda10.

[wanghaoshuang]

复现方法：

• 环境：cuda10 + cudnn7 + python2.7/python3
• 将paddle revert到1.5
• 然后解除单测注释：https://github.com/PaddlePaddle/Paddle/blob/v1.5.2/python/paddle/fluid/tests/unittests/test_warpctc_op.py#L244

执行：

python test_warpctc.py

[Jie-Fang]

I tried to reproduce this bug, and the test successed.
My env:
CUDA10 + cudnn7.2 + python3

Steps:

1. checkout to release/1.5 branch
2. compile
3. add built paddle to PYTHONPATH
4. python3 test_warpctc_op.py

The unittest ran 6 tests.

W1007 03:15:22.005301   434 device_context.cc:259] Please NOTE: device: 0, CUDA Capability: 60, Driver API Version: 10.0, Runtime API Version: 10.0
W1007 03:15:22.007037   434 device_context.cc:267] device: 0, cuDNN Version: 7.2.
......
----------------------------------------------------------------------
Ran 6 tests in 4.791s

OK

Which 7.x cudnn version do you use?

[paddle-bot-old]

Since you haven't replied for more than a year, we have closed this issue/pr.
If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up.
由于您超过一年未回复，我们将关闭这个issue/pr。
若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。