build: Use official nvidia-ml-py package instead of fork

[ecederstrand]

What does this PR address?

There are two packages in PyPI installing a pynvml module - pynvml and nvidia-ml-py. The former is a forked version of the latter, which is the official package published by NVIDIA.

I stumbled upon this because I built a virtual environment that also installs gpustat, and that failed because that package pulls in nvidia-ml-py, thus overwriting the BentoML dependency.

Before submitting:

• Does the Pull Request follow Conventional Commits specification naming? Here are GitHub's
  guide on how to create a pull request.
• Does the code follow BentoML's code style, pre-commit run -a script has passed (instructions)?
• Did you read through contribution guidelines and follow development guidelines?
• Did your changes require updates to the documentation? Have you updated
  those accordingly? Here are documentation guidelines and tips on writting docs.
• Did you write tests to cover your changes?

[sauyon]

Hiya, sorry about the slow response! Mind if I push to fix the CI errors?

[ecederstrand]

Please do 😊

[sauyon]

Ah, seems like there are real test failures caused by the dependency change, we'll probably have to go through and fix those.

[ecederstrand]

Tests are failing because they can't find the NVidia driver (cannot find libnvidia-ml.so.1). I assume the CI pipeline is missing e.g. apt install libnvidia-compute-535-server. Any suggestions on where a good place to add that would be? Also, something similar will be needed for the OS X and Windows tests.

[ecederstrand]

It turns out the tests don't actually test the pynvml package. They just die because pynvml now throws a different exception when being imported. Should be fixed now.

[ecederstrand]

Test failures are dues to a bad pdm.lock file. I'm not familiar with pdm and just wanted to swap pynvml with nvidia-ml-py in pyproject.toml. What's the correct way to do that and re-generate the lock file?

[ecederstrand]

The 3 failing checks don't look related to the changes in this MR.

[aarnphm]

Yep. Our tests are just very much broken.

[aarnphm]

Thanks.