Update GPU detection in merlin.core.utils for Distributed class #98

oliverholworthy · 2022-06-01T16:20:00Z

Updating the automatic GPU detection in merlin.core.utils so that the Distributed class works on both GPU/CPU automatically depending on availability.

Motivation. When adding the XGBoost + Dask integration in merlin models. Added an example to use the merlin.core.utils.Distributed helper and encountered this issue with the default behaviour. NVIDIA-Merlin/models#466

Implementation Details 🚧

A successful import of numba.cuda doesn't necessarily indicate that we have GPUs available.
Adding a check for the length of numba.cuda.gpus.lst, handling a potential CudaSupportError exception.
merlin.core.dispatch has a HAS_GPU variable, however this raises a RuntimeError when the the GPU is unavailable in some configurations. (and a lazy runtime error even if you try to catch a RuntimeError on importing dask_cudf)

Testing Details

Unsure how best to automate tests for this in CI.

Manual tests conducted:

Tested with CUDA available and CUDA_VISIBLE_DEVICES unset -> HAS_GPU = True
Tested with CUDA available but with CUDA_VISIBLE_DEVICES="" -> HAS_GPU = False
Tested with CUDA unavailable using docker without --gpu setting -> HAS_GPU = False

github-actions · 2022-06-01T16:30:20Z

Documentation preview

https://nvidia-merlin.github.io/core/review/pr-98

nvidia-merlin-bot · 2022-06-01T16:43:47Z

Click to view CI Results

GitHub pull request #98 of commit 5386077a55d023523a735ba2d21a9a3be18685ed, no merge conflicts.
Running as SYSTEM
Setting status of 5386077a55d023523a735ba2d21a9a3be18685ed to PENDING with url https://10.20.13.93:8080/job/merlin_core/65/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/98/*:refs/remotes/origin/pr/98/* # timeout=10
 > git rev-parse 5386077a55d023523a735ba2d21a9a3be18685ed^{commit} # timeout=10
Checking out Revision 5386077a55d023523a735ba2d21a9a3be18685ed (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5386077a55d023523a735ba2d21a9a3be18685ed # timeout=10
Commit message: "Check cuda.gpus.lst for available GPUs"
 > git rev-list --no-walk f336ca3ff96810efbded64e0559ebb880ee06364 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins14859643852832795028.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already up-to-date: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (62.3.2)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 342 items / 1 skipped
tests/unit/core/test_dispatch.py ..                                      [  0%]

tests/unit/dag/test_base_operator.py ....                                [  1%]

tests/unit/dag/test_column_selector.py ..........................        [  9%]

tests/unit/dag/test_tags.py ......                                       [ 11%]

tests/unit/dag/ops/test_selection.py ...                                 [ 11%]

tests/unit/io/test_io.py ............................................... [ 25%]

................................................................         [ 44%]

tests/unit/schema/test_column_schemas.py ............................... [ 53%]

........................................................................ [ 74%]

.......................................................................  [ 95%]

tests/unit/schema/test_schema.py ......                                  [ 97%]

tests/unit/schema/test_schema_io.py ..                                   [ 97%]

tests/unit/utils/test_utils.py ........                                  [100%]
=============================== warnings summary ===============================

tests/unit/dag/test_base_operator.py: 4 warnings

tests/unit/io/test_io.py: 72 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1292: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]
tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 35585 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44655 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 45205 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 40287 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 36041 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33023 instead

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

================= 342 passed, 1 skipped, 83 warnings in 53.79s =================

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins8223533369104201474.sh

nvidia-merlin-bot · 2022-06-01T16:57:44Z

Click to view CI Results

GitHub pull request #98 of commit 21fbb71aa1808f19a1357651992b0c2f9bb60239, no merge conflicts.
Running as SYSTEM
Setting status of 21fbb71aa1808f19a1357651992b0c2f9bb60239 to PENDING with url https://10.20.13.93:8080/job/merlin_core/66/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/98/*:refs/remotes/origin/pr/98/* # timeout=10
 > git rev-parse 21fbb71aa1808f19a1357651992b0c2f9bb60239^{commit} # timeout=10
Checking out Revision 21fbb71aa1808f19a1357651992b0c2f9bb60239 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 21fbb71aa1808f19a1357651992b0c2f9bb60239 # timeout=10
Commit message: "Check cuda.gpus.lst for available GPUs"
 > git rev-list --no-walk 5386077a55d023523a735ba2d21a9a3be18685ed # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins15247958366670095269.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already up-to-date: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (62.3.2)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 342 items / 1 skipped
tests/unit/core/test_dispatch.py ..                                      [  0%]

tests/unit/dag/test_base_operator.py ....                                [  1%]

tests/unit/dag/test_column_selector.py ..........................        [  9%]

tests/unit/dag/test_tags.py ......                                       [ 11%]

tests/unit/dag/ops/test_selection.py ...                                 [ 11%]

tests/unit/io/test_io.py ............................................... [ 25%]

................................................................         [ 44%]

tests/unit/schema/test_column_schemas.py ............................... [ 53%]

........................................................................ [ 74%]

.......................................................................  [ 95%]

tests/unit/schema/test_schema.py ......                                  [ 97%]

tests/unit/schema/test_schema_io.py ..                                   [ 97%]

tests/unit/utils/test_utils.py ........                                  [100%]
=============================== warnings summary ===============================

tests/unit/dag/test_base_operator.py: 4 warnings

tests/unit/io/test_io.py: 72 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1292: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]
tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42485 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 44075 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 34629 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42443 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 37991 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42329 instead

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

================= 342 passed, 1 skipped, 83 warnings in 55.90s =================

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins18090326885695540167.sh

merlin/core/utils.py

nvidia-merlin-bot · 2022-06-01T17:23:49Z

Click to view CI Results

GitHub pull request #98 of commit 0e6f300caea67715418756e1f77b3990d8010caf, no merge conflicts.
Running as SYSTEM
Setting status of 0e6f300caea67715418756e1f77b3990d8010caf to PENDING with url https://10.20.13.93:8080/job/merlin_core/67/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_core
using credential ce87ff3c-94f0-400a-8303-cb4acb4918b5
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/core # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/core
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems username and pass
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/core +refs/pull/98/*:refs/remotes/origin/pr/98/* # timeout=10
 > git rev-parse 0e6f300caea67715418756e1f77b3990d8010caf^{commit} # timeout=10
Checking out Revision 0e6f300caea67715418756e1f77b3990d8010caf (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e6f300caea67715418756e1f77b3990d8010caf # timeout=10
Commit message: "Move gpu check to compat module"
 > git rev-list --no-walk 21fbb71aa1808f19a1357651992b0c2f9bb60239 # timeout=10
[merlin_core] $ /bin/bash /tmp/jenkins13254012742949887097.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already up-to-date: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (62.3.2)
/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pypi.ngc.nvidia.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_core/core, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 342 items / 1 skipped
tests/unit/core/test_dispatch.py ..                                      [  0%]

tests/unit/dag/test_base_operator.py ....                                [  1%]

tests/unit/dag/test_column_selector.py ..........................        [  9%]

tests/unit/dag/test_tags.py ......                                       [ 11%]

tests/unit/dag/ops/test_selection.py ...                                 [ 11%]

tests/unit/io/test_io.py ............................................... [ 25%]

................................................................         [ 44%]

tests/unit/schema/test_column_schemas.py ............................... [ 53%]

........................................................................ [ 74%]

.......................................................................  [ 95%]

tests/unit/schema/test_schema.py ......                                  [ 97%]

tests/unit/schema/test_schema_io.py ..                                   [ 97%]

tests/unit/utils/test_utils.py ........                                  [100%]
=============================== warnings summary ===============================

tests/unit/dag/test_base_operator.py: 4 warnings

tests/unit/io/test_io.py: 72 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1292: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(
tests/unit/io/test_io.py::test_validate_and_regenerate_dataset

/var/jenkins_home/workspace/merlin_core/core/merlin/io/parquet.py:551: DeprecationWarning: 'ParquetDataset.pieces' attribute is deprecated as of pyarrow 5.0.0 and will be removed in a future version. Specify 'use_legacy_dataset=False' while constructing the ParquetDataset, and then use the '.fragments' attribute instead.

paths = [p.path for p in pa_dataset.pieces]
tests/unit/utils/test_utils.py::test_nvt_distributed[True-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 35039 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[True-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38375 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[False-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 42605 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed[False-False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 41803 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed_force[True]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 38909 instead

warnings.warn(
tests/unit/utils/test_utils.py::test_nvt_distributed_force[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:160: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 40875 instead

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

================= 342 passed, 1 skipped, 83 warnings in 56.00s =================

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/core/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_core] $ /bin/bash /tmp/jenkins579037059933515431.sh

oliverholworthy changed the title ~~Use HAS_GPU from merlin.core.dispatch in Distributed class~~ Update GPU detection in merlin.core.utils for Distributed class Jun 1, 2022

Check cuda.gpus.lst for available GPUs

21fbb71

oliverholworthy force-pushed the distributed-utils-use-has-gpu branch from 5386077 to 21fbb71 Compare June 1, 2022 16:56

oliverholworthy mentioned this pull request Jun 1, 2022

XGBoost - Switch to dask API NVIDIA-Merlin/models#466

Merged

oliverholworthy requested review from karlhigley and benfred June 1, 2022 16:58

benfred reviewed Jun 1, 2022

View reviewed changes

merlin/core/utils.py Outdated Show resolved Hide resolved

Move gpu check to compat module

0e6f300

karlhigley approved these changes Jun 1, 2022

View reviewed changes

karlhigley merged commit 136f688 into NVIDIA-Merlin:main Jun 1, 2022

oliverholworthy mentioned this pull request Jun 7, 2022

Update imports in dispatch module to handle missing GPU #99

Merged

oliverholworthy added the bug Something isn't working label Jun 8, 2022

rjzamora mentioned this pull request Oct 4, 2022

[Bug] Using numba to detect gpu availability breaks Dask-CUDA worker pinning #144

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update GPU detection in merlin.core.utils for Distributed class #98

Update GPU detection in merlin.core.utils for Distributed class #98

oliverholworthy commented Jun 1, 2022 •

edited

github-actions bot commented Jun 1, 2022

nvidia-merlin-bot commented Jun 1, 2022

nvidia-merlin-bot commented Jun 1, 2022

nvidia-merlin-bot commented Jun 1, 2022

Update GPU detection in merlin.core.utils for Distributed class #98

Update GPU detection in merlin.core.utils for Distributed class #98

Conversation

oliverholworthy commented Jun 1, 2022 • edited

Implementation Details 🚧

Testing Details

github-actions bot commented Jun 1, 2022

Documentation preview

nvidia-merlin-bot commented Jun 1, 2022

nvidia-merlin-bot commented Jun 1, 2022

nvidia-merlin-bot commented Jun 1, 2022

oliverholworthy commented Jun 1, 2022 •

edited