Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

Closed
BEpresent opened this issue May 4, 2023 · 17 comments
Closed

module 'pynvml' has no attribute '_nvmlGetFunctionPointer' #153

BEpresent opened this issue May 4, 2023 · 17 comments
Milestone

Comments

@BEpresent
Copy link

Describe the bug

Calling gpustat leads to the following error:

Screenshots or Program Output

Traceback (most recent call last):
  File "/home/be/.local/bin/gpustat", line 5, in <module>
    from gpustat import main
  File "/home/be/.local/lib/python3.9/site-packages/gpustat/__init__.py", line 16, in <module>
    from .core import GPUStat, GPUStatCollection
  File "/home/be/.local/lib/python3.9/site-packages/gpustat/core.py", line 24, in <module>
    from gpustat.nvml import pynvml as N
  File "/home/be/.local/lib/python3.9/site-packages/gpustat/nvml.py", line 57, in <module>
    _original_nvmlGetFunctionPointer = pynvml._nvmlGetFunctionPointer
AttributeError: module 'pynvml' has no attribute '_nvmlGetFunctionPointer'

Please provide the output of gpustat --debug and nvidia-smi. Or attach screenshots if applicable.

image

Environment information:

  • OS: Debian 11.7
  • NVIDIA Driver version: NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.4
  • The name(s) of GPU card: 2 T4
  • gpustat version: gpustat --version doesn't work, but I did pip install it so I assume it fetches the latest version Successfully uninstalled gpustat-1.1
  • pynvml version: nvidia-ml-py 11.525.112

Additional context

It should be noted that it has worked 2-3 weeks ago, but now it does not on 3 different machines I have tested.

@BEpresent BEpresent added the bug label May 4, 2023
@wookayin
Copy link
Owner

wookayin commented May 7, 2023

Thanks for the report. Hmm, I see _nvmlGetFunctionPointer still exists in 11.525.112 version:

https://github.com/wookayin/nvidia-ml-py/blob/11.525.112/pynvml.py#L853

Could you please try opening the file /home/be/.local/lib/python3.9/site-packages/pynvml.py and see if the function exists? It should be at around Line 852.

@herrenP

This comment was marked as duplicate.

@herrenP
Copy link

herrenP commented May 10, 2023

This fixed my issue at least:

~ > pip install --force-reinstall gpustat
Defaulting to user installation because normal site-packages is not writeable
Collecting gpustat
  Downloading gpustat-1.1.tar.gz (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 2.0 MB/s eta 0:00:00
...removed lines for brevity...
Successfully installed blessed-1.20.0 gpustat-1.1 nvidia-ml-py-11.525.112 psutil-5.9.5 six-1.16.0 wcwidth-0.2.6

~ > gpustat                                                                                     
[0] NVIDIA GeForce RTX 3070 | 52°C,  21 % |   553 /  8192 MB | ...

@wookayin
Copy link
Owner

wookayin commented May 10, 2023

TL;DR) force-reinstall nvidia-ml-py; pip install --force-reinstall --ignore-installed nvidia-ml-py

Thank you for the information. Yes, the pynvml should be the one that you get from your site-packages:

$ python -c 'import pynvml; print(pynvml.__file__)'
/usr/lib/python3.10/site-packages/pynvml.py

So apparently this pynvml seems to be coming from an old version of pynvml. This can happen, for instance, some other python packages install legacy nvidia-ml-py3 which might override pynvml; pip list can't tell this. So that's why I'm asking the checksum of the pynvml file to ensure the package hasn't been corrupted.

Although this is an user problem (or one of the downstream packages used in your project), I will try to add some fool-proof error messages when pynvml isn't alright.

@mattip

This comment was marked as off-topic.

@wookayin
Copy link
Owner

wookayin commented May 17, 2023

According to ray-project/ray#34196 (/cc @yuanwu2017), the problematic pynvml file comes from this third-party fork(?):

Actually this is published nowhere so I have no idea why users end up having this conflicting package. https://pypi.org/project/pynvml/ this should not be used.

@mattip
Copy link
Contributor

mattip commented May 17, 2023

Hmm. Is there any way to distinguish between nvidia-ml-py and pynvml, and perhaps raise a helpful error or somehow integrate the two APIs? Looking at the download stats, pynvml has about 113,500 / week, nvidia-ml-py has about 390,500 / week, so there are a lot of people using pynvml. I can understand them, anyone looking for the "natural" project to supply pynvml would assume the project name is pynvml not nvidia-ml-py.

@wookayin
Copy link
Owner

wookayin commented May 17, 2023

nvidia-ml-py is the only official, right package to use. But it's also likely that some third-party downstream packages use the wrong one. It's possible to detect which one is used so I will try to add some warning messages. Note that this error is happening due to the monkey-patching used in #143 which are trying to monkey-patch a private API of pynvml; the gpuopenanalytic's pynvml package mostly works the similar is not exposing private APIs. (I think this package should not have existed if they are going to provide the same API...)

@mattip
Copy link
Contributor

mattip commented May 17, 2023

here are the packages github detects as depending on pynvml. There are 156, some with over 1000 stars. I opened issues on the ones with lots of stars. 😄

@wookayin
Copy link
Owner

wookayin commented May 17, 2023

I've added some error messages. It would be not difficult to make gpustat compatible with the non-official pynvml package, but let me figure out a good solution when I have more spare time.

@wookayin wookayin pinned this issue May 17, 2023
wookayin added a commit that referenced this issue Jun 1, 2023
An un-official distribution of the pynvml package provided by
@gpuopenanalytics/pynvml was causing an incompatibility issue, where
gpustat.nvml fails to import. Although the official and correct pynvml
packages should be installed, but wild users' python environments can be
easily messed up by unawaringly installing the problematic, unofficial
pynvml package.

gpustat can support this third-party pynvml by directly importing the
low-level pynvml.nvml package (which should be exactly the same as
an official pynvml.py). This is a quickfix that relies on the
implementation detail of a third-party package, but nevertheless it
should be a good enough workaround solution for many users in practice.
@wookayin
Copy link
Owner

wookayin commented Jun 1, 2023

7c09a0f makes gpustat compatible even with @gpuopenanalytics/pynvml. Closing this as fixed.

@wookayin wookayin closed this as completed Jun 1, 2023
@sam-my-2023

This comment was marked as duplicate.

@wookayin
Copy link
Owner

You'll also need to do pip uninstall pynvml as well.

@BEpresent

This comment was marked as duplicate.

@zhiwenwang20122302

This comment was marked as duplicate.

@cdj0311

This comment was marked as duplicate.

wookayin added a commit that referenced this issue Aug 22, 2023
(cherry picked from commit e6bf58f,
 backport to 1.1.x)
wookayin added a commit that referenced this issue Aug 22, 2023
An un-official distribution of the pynvml package provided by
@gpuopenanalytics/pynvml was causing an incompatibility issue, where
gpustat.nvml fails to import. Although the official and correct pynvml
packages should be installed, but wild users' python environments can be
easily messed up by unawaringly installing the problematic, unofficial
pynvml package.

gpustat can support this third-party pynvml by directly importing the
low-level pynvml.nvml package (which should be exactly the same as
an official pynvml.py). This is a quickfix that relies on the
implementation detail of a third-party package, but nevertheless it
should be a good enough workaround solution for many users in practice.

(cherry picked from commit 7c09a0f,
 backported to 1.1.x)
@wookayin
Copy link
Owner

(backported to v1.1.1 and released)

@wookayin wookayin added this to the 1.2 milestone Aug 22, 2023
wookayin added a commit that referenced this issue Jan 12, 2024
Revert "Make gpustat.nvml compatible with a third-party fork of pynvml"

This reverts commit 7c09a0f.

gpustat v1.1.1 allowed the problematic 'pynvml' package to be used
as a workaround, but this still causes many problems (e.g., #168).
Only the official nvidia-ml-py can be used with gpustat.

See #153, #168
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants