cuDNN convolution try multiple algo

[zasdfgbnm]

Fixes: #31336 #1664

Sometimes cuDNN heuristics return algorithms that can not be used. Instead of just using the first algorithm returned, we should try these algorithms one by one until one of them succeed.

Benchmark:
https://github.com/zasdfgbnm/things/blob/master/2020Q1/conv-benchmark.ipynb

i = torch.randn(256, 3, 256, 256).cuda()
c = torch.nn.Conv2d(3, 3, 3, 3).cuda()

%timeit c(i); torch.cuda.synchronize()

before vs after = 498 vs 490 µs

The performance is improved I guess because, before this PR, we always call the heuristics to get the algorithm, but after this PR, we only do at the first time.

[dr-ci]

💊 CircleCI build failures summary and remediations

As of commit 16cf1ff (more details on the Dr. CI page):

---

✅ None of the build failures appear to be your fault 💚

---

• 7/7 broken upstream at merge base ac6e75a since Mar 04

  Please rebase on the viable/strict branch (expand for instructions)

  If your commit is newer than viable/strict, you can try basing on an older, stable commit:

  git fetch origin viable/strict
  git rebase --onto viable/strict $(git merge-base origin/master HEAD)
  

  If your commit is older than viable/strict:

  git fetch origin viable/strict
  git rebase viable/strict
  

  Check out the recency history of this "viable master" tracking branch.

---

🚧 7 upstream failures recognized by patterns:

These were probably caused by upstream breakages:

• pytorch_linux_xenial_cuda10_1_cudnn7_py3_multigpu_test from Mar 04 until Mar 04
• pytorch_linux_xenial_cuda10_1_cudnn7_py3_slow_test from Mar 04 until Mar 04
• pytorch_linux_xenial_cuda10_1_cudnn7_py3_nogpu_test from Mar 04 until Mar 04
• pytorch_xla_linux_xenial_py3_6_clang7_test from Mar 04
• pytorch_linux_xenial_cuda10_1_cudnn7_py3_NO_AVX_NO_AVX2_test from Mar 04 until Mar 04
• pytorch_linux_xenial_cuda10_1_cudnn7_py3_gcc7_test from Mar 04 until Mar 04
• pytorch_linux_xenial_cuda10_1_cudnn7_py3_NO_AVX2_test from Mar 04 until Mar 04

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 25 times.

[zasdfgbnm]

cc: @ptrblck @csarofeen

[zasdfgbnm]

This also seems to fix #31500

[zasdfgbnm]

This would conflict with #33056

[zasdfgbnm]

Ahha, this is conflicting

[ngimel]

LGTM

[facebook-github-bot]

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@ngimel merged this pull request in e4a883e.