[ci] bump CUDA version and pull dockers from NVIDIA NGC

[StrikerRUS]

Why switch to NVIDIA NGC? Due to #3518.

[jameslamb]

ugh I'm surprised to see this error from #4057 again (error message was introduced in #4059)

https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=9565&view=logs&j=c28dceab-947a-5848-c21f-eef3695e5f11&t=fa158246-17e2-53d4-5936-86070edbaacf

machines = '127.0.0.1:53229,127.0.0.1:53229'
worker_addresses = dict_keys(['tcp://127.0.0.1:43843', 'tcp://127.0.0.1:42213'])

    def _machines_to_worker_map(machines: str, worker_addresses: List[str]) -> Dict[str, int]:
        """Create a worker_map from machines list.
    
        Given ``machines`` and a list of Dask worker addresses, return a mapping where the keys are
        ``worker_addresses`` and the values are ports from ``machines``.
    
        Parameters
        ----------
        machines : str
            A comma-delimited list of workers, of the form ``ip1:port,ip2:port``.
        worker_addresses : list of str
            A list of Dask worker addresses, of the form ``{protocol}{hostname}:{port}``, where ``port`` is the port Dask's scheduler uses to talk to that worker.
    
        Returns
        -------
        result : Dict[str, int]
            Dictionary where keys are work addresses in the form expected by Dask and values are a port for LightGBM to use.
        """
        machine_addresses = machines.split(",")
    
        if len(set(machine_addresses)) != len(machine_addresses):
>           raise ValueError(f"Found duplicates in 'machines' ({machines}). Each entry in 'machines' must be a unique IP-port combination.")
E           ValueError: Found duplicates in 'machines' (127.0.0.1:53229,127.0.0.1:53229). Each entry in 'machines' must be a unique IP-port combination.

[StrikerRUS]

I have already seen this error one time after #4059. Guess this is somehow related to the initial seed of pseudo-random generator.
I believe more fair test from #4057 (comment) will be to run lgb.dask._find_random_open_port() with Python restart.

[jameslamb]

oh really good idea to switch to NGC, thanks

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.