[1.9.1] [collect_env] collect_env does not collect actual runtime-loaded cudnn version

[vadimkantorov]

🐛 Describe the bug

In opposite, torch.backends.cudnn.version() does collect it correctly.
Originally reported by @grazder in a comment: #78475 (comment)

This occurred on old-ish version of 1.9.1, not sure if it still happens on 1.11.0

Versions

1.9.1

cc @csarofeen @ptrblck @xwang233

[malfet]

collect_env collects version of cudnn installed on the system, rather than the one PyTorch were compiled against (similarly to CUDA used to build PyTorch: vs CUDA runtime version:)

[ptrblck]

Just wanted to post the same :P
Reference can be seen here.

[vadimkantorov]

I guess it is what's happening. But for debugging it makes sense to also report the version linked in / discovered by PyTorch.

A lazy version (if PyTorch installation is workable enough to be imported) could just report this torch.backends.cudnn.version(). Some less demanding version could probably scan the binaries if cudnn linked in statically or even copying contents of some build-time-generated versions.py distributed with PyTorch that lists linked libraries versions / compilation / linking config

As is, it's quite confusing, since requires to manually find linked in versions, since they are quite relevant

Bonus for making the printed message clearer to explain what exactly collect_env is trying to discover/report (version at build time, version at linking time, dynamically loaded version, etc)

Reporting both env AND actual pytorch found one is important for debugging some linking/loading issues when PyTorch is loading a version different from what it is compiled against

[vadimkantorov]

Related: #80637

[vadimkantorov]

So probably can be done by reading f"/proc/{os.getpid()}/maps" (on Unix machines, and if this file exists) and filtering cudnn/blas/actually loaded dependencies .so files

[vadimkantorov]

Also, there is some improved collect_env in detectron2: https://github.com/facebookresearch/detectron2/blob/main/detectron2/utils/collect_env.py#L55