[nccl] Remove NCCL auto-download due to licensing issues

[jacobkahn]

• Pursuant to Update VMs, CMake to 3.20.1, CUDA to 11.3, and pwsh to 7.1.3 #17331, remove NCCL auto-download due to licensing issues.

• Also improves finding NCCL - uses common environment variables

• What does your PR fix?

Follow up to #16031, mimics #16413, and makes additional changes as outlined in #17346

Fixes flashlight/wav2letter#950

• Which triplets are supported/not supported? Have you updated the CI baseline?

Same triplets as before. This doesn't affect the CI baseline since CUDA 10.1 is still installed on CI Machines and NCCL 2.4.6 (compatible with 10.1) will still be downloaded from conda.

• Does your PR follow the maintainer guide?

Yes

• If you have added/updated a port: Have you run ./vcpkg x-add-version --all and committed the result?

Yes

[cenit]

please consider this:
#17331 (comment)

[jacobkahn]

Aha - thanks for flagging @cenit -- I'll go ahead and remove it here too for this PR. cc @BillyONeal — if you want to tack NCCL onto your request, that would be helpful!

Too bad that we have to remove it, but makes sense.

[BillyONeal]

Will have this pending shortly:

[PhoebeHui]

Depends on PR #17331.

Failures on linux:

-- Could NOT find NCCL: Found unsuitable version "", but required is at least "2.4.6.1" (found NCCL_INCLUDE_DIRS-NOTFOUND)
CMake Error at ports/nccl/portfile.cmake:16 (message):
  Please install NCCL using your system package manager (the same way you
  installed CUDA).  For example: apt install libnccl2 libnccl-dev.
Call Stack (most recent call first):
  scripts/ports.cmake:142 (include)

[PhoebeHui]

@jacobkahn, thanks for your contribution!

Could you update the baseline version by executing ' ./vcpkg x-add-version --overwrite-version nccl'?

Error: While reading versions for port nccl from file: C:\a\1\s\versions\n-\nccl.json
       File declares version `2.4.6#1` with SHA: 78f69d23214c0c0849fbe1a5d0fe62f934dc72e1
       But local port with the same verion has a different SHA: 65d03efc8a527afe223125d2cac2e685517297e3
       Please update the port's version fields and then run:

           vcpkg x-add-version nccl

       to add a new version.

[jacobkahn]

@PhoebeHui done.

I think CI needs a rerun now that NCCL is installed.

[BillyONeal]

I think CI needs a rerun now that NCCL is installed.

You can always trigger that by merging with origin/master.

[BillyONeal]

Looks OK to me assuming CI passes.

[BillyONeal]

Looks like CI failures are a regression from #13100 + updating the compilers on the build machines to 16.9.4

[PhoebeHui]

Yes, For gdk-pixbuf's failure, it's not relate to this PR, we would take the baseline error.

Failures:

E:\vcpkg\clean\vcpkg\buildtrees\gdk-pixbuf\src\k-pixbuf-2-c86079bed6\gdk-pixbuf\pixops\../fallback-c89.c(38): error C2169: 'lrint': intrinsic function, cannot be defined

For nccl:x64-linux, it still failed with following failure:

-- Could NOT find NCCL: Found unsuitable version "", but required is at least "2.4.6.1" (found NCCL_INCLUDE_DIRS-NOTFOUND)
CMake Error at ports/nccl/portfile.cmake:16 (message):
  Please install NCCL using your system package manager (the same way you
  installed CUDA).  For example: apt install libnccl2 libnccl-dev.
Call Stack (most recent call first):
  scripts/ports.cmake:142 (include)

[Neumann-A]

gdk-pixbuf check for lrint in meson.build needs to be updated to use the correct parameters. I often see function checks in the form of somefunction(void) and VS is eagerly reporting errors with release build flags while with debug build flags the check passes fine.

[jacobkahn]

Looks like the dependent PR has merged -- are we good here @BillyONeal @PhoebeHui ?

[PhoebeHui]

@jacobkahn, I have rerun the CI testing, let's wait for the results.

[BillyONeal]

Looks like the dependent PR has merged -- are we good here @BillyONeal @PhoebeHui ?

This one looks like the port actually not being happy:

-- Could NOT find NCCL: Found unsuitable version "", but required is at least "2.4.6.1" (found NCCL_INCLUDE_DIRS-NOTFOUND)
CMake Error at ports/nccl/portfile.cmake:16 (message):
  Please install NCCL using your system package manager (the same way you
  installed CUDA).  For example: apt install libnccl2 libnccl-dev.
Call Stack (most recent call first):
  scripts/ports.cmake:142 (include)

We installed nccl into our base VMs like this:

vcpkg/scripts/azure-pipelines/linux/provision-image.sh

Line 66 in 1efdaf8

sudo apt install -y --no-install-recommends cuda-compiler-11-3 cuda-libraries-dev-11-3 cuda-driver-dev-11-3 \

No longer relevant

[jacobkahn]

@BillyONeal reprod and fixed. It seems that like cuDNN, NCCL throws bits directly in /usr/include and /usr/lib, and there's no pkgconfig, so added those hints as we have for the cuDNN package.

[strega-nil-ms]

Thanks @jacobkahn and @BillyONeal !

[longnguyen2004]

Can we also build nccl from source? The source is at https://github.com/NVIDIA/nccl/tree/master/src

[BillyONeal]

@longnguyen2004 I would really like to get some form of nvidia comment before going there; the licensing in that repo is not consistent.

[longnguyen2004]

Seems to me that it's mostly BSD 3-clause, judging from this page https://docs.nvidia.com/deeplearning/nccl/bsd/index.html