Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPU monitoring (AMD / ATI) #993

Closed
nicolargo opened this issue Jan 8, 2017 · 36 comments
Closed

GPU monitoring (AMD / ATI) #993

nicolargo opened this issue Jan 8, 2017 · 36 comments

Comments

@nicolargo
Copy link
Owner

nicolargo commented Jan 8, 2017

Description

As implemented in the issue #170 for the NVidia GPUs, the goal is to enhance the GPU plugin for AMD / ATI GPUs.

First of all, we have to find a Python Lib to grab stats, then change the current plugin to be able to monitor AMD / ATI GPU.

@kdbanman
Copy link
Contributor

kdbanman commented Jan 8, 2017

As shared in #994, this and this were the starting points of my research.


First of all, we have to find a Python Lib to grab stats

I don't know of any python libraries that do this out of the box. But RadeonTop and aticonfig are command line tools that both grab stats.

They can be used like this to get stats. That code spawns a subprocess running aticonfig and parses its text output. That technique may not be a good long term solution though.

@nicolargo nicolargo modified the milestone: Glances 2.9 Jan 21, 2017
@nicolargo nicolargo modified the milestone: Glances 2.9 Mar 10, 2017
@nicolargo
Copy link
Owner Author

Also have a look on https://github.com/asornoso/AMD-GPU-INFO

@hunasdf
Copy link

hunasdf commented Dec 20, 2017

Amazing job guys! I miss this AMD/ATI GPU monitoring plugin only. This may can help. Old code, but I hope to you can use it :)
https://github.com/bitshiftio/pyADL or https://github.com/mjmvisser/adl3

@nicolargo
Copy link
Owner Author

@hunasdf Nice ! I will have a look on it and came back to you for the testing step because i do not have any AMD/ATI device...

@nicolargo nicolargo added this to the Glances 3.0 milestone Dec 21, 2017
@hunasdf
Copy link

hunasdf commented Dec 21, 2017

@nicolargo Of course, I help for you! I am a programer too with a little Python knowledge (i am learning it). And I have 5 RX 460 4GB in a single machine and 1 R9 380X 2GB card in another one. On both machine running Win 10 now. Just write to me to what I should do :)

@nicolargo
Copy link
Owner Author

nicolargo commented Dec 21, 2017

@hunasdf Just have a quick look on the ADL3 lib (available in Pypi). No documentation and the code is very old...

Can you just try this on your machine (from a console) and copy/paste the result:

pip install adl3
python
>>> import adl3
>>> dir(adl3)

Thanks

Note, on my Linux machine without AMD/ATI card:

In [2]: import adl3
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-50ec25a0a1b3> in <module>()
----> 1 import adl3

/usr/local/lib/python2.7/dist-packages/adl3/__init__.py in <module>()
----> 1 from .adl_api import *

/usr/local/lib/python2.7/dist-packages/adl3/adl_api.py in <module>()
     40 
     41         # load the ADL 3.0 dso/dll
---> 42         _libadl = CDLL("libatiadlxx.so", mode=RTLD_GLOBAL)
     43 
     44         # ADL requires we pass an allocation function and handle freeing it ourselves

/usr/lib/python2.7/ctypes/__init__.pyc in __init__(self, name, mode, handle, use_errno, use_last_error)
    360 
    361         if handle is None:
--> 362             self._handle = _dlopen(self._name, mode)
    363         else:
    364             self._handle = handle

OSError: libatiadlxx.so: cannot open shared object file: No such file or directory

@hunasdf
Copy link

hunasdf commented Dec 22, 2017

@nicolargo On Win 10, with R9 380x (python 2.7)
But I can add access for you for this machine if it is help :)

adl3.txt

@nicolargo
Copy link
Owner Author

@hunasdf Yes it will be nice but without any documentation it will not be easy...

@hunasdf
Copy link

hunasdf commented Dec 22, 2017

@nicolargo I check this lib and I do not know... Maybe we can use a part of it.
I tried to use "atitweak -l" for simple list the adapters and I got an error. No problem, I started to debug the issue. I found the mistake at ADL_Adapter_ID_Get (ADL_Adapter_NumberOfAdapters_Get and ADL_Adapter_AdapterInfo_Get maybe OK). I started to search any reference or something about it, and I found this page: https://developer.amd.com/display-library-adl-sdk/ The latest ADL version is 10 (vs 3 what i tried to use). I downloaded it and the zip contains full documentation about it. After a little search i found ADL2_Adapter_Active_Get. So the function is renamed and has other parameter list too.
As a result, I have some result and if I have a little time, I will continue the debugging/developing.

Ps.: Sorry for my English, I hope you understand it :)

@nicolargo
Copy link
Owner Author

@hunasdf thanks for your time and do not worry about your English ;)

I want to let specific plugin code outside the Glances source project. So i prefer to have an external ATI/AMD Python Lib. If you want i can initialize a "Git repository skeleton" for this new lib where you test your development. What do you think ?

@hunasdf
Copy link

hunasdf commented Dec 22, 2017

It is a great idea, lets try it!

@hunasdf
Copy link

hunasdf commented Dec 23, 2017

@nicolargo Good news: something happened!
I updated the ATI/AMD driver to 17.12 (block chain driver), and restarted the machine. And use ADL2_Adapter_ID_Get instead of ADL_Adapter_ID_Get. Now, here the output:

adl3-master>python atitweak -l
0. AMD Radeon (TM) R9 380 Series (\.\DISPLAY4)
engine clock range is 150 - 1200MHz
memory clock range is 75 - 1750MHz
core voltage range is 0 - 0VDC
performance level 0: engine clock 300MHz, memory clock 150MHz, core voltage 0VDC
performance level 1: engine clock 985MHz, memory clock 1400MHz, core voltage 0VDC
fan speed range: 0 - 100%, 0 - 6000 RPM

@nicolargo
Copy link
Owner Author

nicolargo commented Dec 24, 2017

@hunasdf As already done for NVidia (see http://glances.readthedocs.io/en/stable/aoa/gpu.html) the Glances GPU plugin should display the following information for the AMD/ATI GPU:

  1. GPU name ==> Look like the information is available ("0. AMD Radeon (TM) R9 380 Series (.\DISPLAY4)). I think that the "0." is the GPU number ?
  2. GPU process load in %. I do not see this stat in the output of the atitweak command line. Perhaps the "engine clock range is 150 - 1200MHz" (150*100/1200) ? What the difference with the performance level line ?
  3. Memory consumption in %. I do not see it ? Is there another method to grab this stat from the ADL lib ?
  4. Additionally, we can also the FAN speed in the Sensor plugin (what about the temperature sensor ?)

@hunasdf I just create a new Github repository with the skeleon for the future lib. I call it PyADL (https://github.com/nicolargo/pyadl). Please click here to be added as collaborator. You can now clone, commit and push your dev inside. Let me know if you need additional information.

@hunasdf
Copy link

hunasdf commented Dec 24, 2017

@nicolargo I think, this informations is available in the ADL. I seen somewhere. Next step will be grab there. (The infos in my last comment were the same output what atitweak generated. I just was happy to we can use this “old” adl3 lib :))
And thanks the invitation, I accepted it!

@hunasdf
Copy link

hunasdf commented Dec 26, 2017

I commited the first alpha version. With this, you can grab the described infos except Memory consumption. If something not okay with the code, the license, anything, just write to me! :)

@nicolargo
Copy link
Owner Author

Hi @hunasdf

My comments here: nicolargo/pyadl#1

Thanks !

@nicolargo
Copy link
Owner Author

@hunasdf Any news concerning PyADL ? Do you think that it will be available soon ? If not i have to postpone the implementation of the Glances AMD GPU... Let me know.

@hunasdf
Copy link

hunasdf commented Feb 4, 2018

@nicolargo I did not deal with it (i did not have enough time). In my opinion, we must to find other solution for linux. The latest ADL SDK came out on "08/10/2016", so .... I do not happy about it, but yes, we have to postpone the integration :(
What do you think, it is a problem to we will use this PyADL for windows and a totally other solution for linux? For example sysfs.

@nicolargo
Copy link
Owner Author

From a Glances dependencies point of view it is a problem because it is a cross platform software so the difference between Windows and others OS should be manage by the lib (PyADL), not by Glances.

@nicolargo nicolargo removed this from the Glances 3.0 milestone Feb 6, 2018
@hunasdf
Copy link

hunasdf commented Feb 6, 2018

Of course, PyADL should manage it. But if PyADL uses something other, its name can be deception, maybe.

@douglasg14b
Copy link

douglasg14b commented Jul 25, 2019

@nicolargo For GPU temps, it seems the normal systems input for telegraf pulls ATI GPU Temp sensors as well, if that's some sort of hint of where you could grab that info.

@nicolargo nicolargo added this to the Glances 4.0 milestone Nov 16, 2019
@kdschlosser
Copy link

I know this is an older issue but I thought I would share

https://github.com/kdschlosser/ati_radeon

@johntiger1
Copy link

Is GPU monitoring for AMD gpus implemented yet? Thanks

@nicolargo
Copy link
Owner Author

@johntiger1 Nope because there is no cross platform Python lib to grab GPU AMD stat correctly.

@imdebating
Copy link

Apologies for helicoptering into this issue...Python is not my thing. I grabbed a couple of Z600s that came with old AMD cards, so started searching to see if Glances could support it and ended up here.
Because this issue is old, I did a search on PyPI and found pyamdgpuinfo, which appears to be an active (in beta) project. No idea if this helps or not, but thought I'd share.
As always, thank you @nicolargo for this amazing app.

@nicolargo
Copy link
Owner Author

@imdebating https://pypi.org/project/pyamdgpuinfo/ lloks geat ! I will have a look :) Thanks for the head up !

@nicolargo
Copy link
Owner Author

@PhilipDeegan
Copy link

PhilipDeegan commented Mar 26, 2024

@nicolargo nvtop is nice https://news.ycombinator.com/item?id=39687132

works for amd too (or for me at least)

image

@nicolargo
Copy link
Owner Author

nicolargo commented Apr 2, 2024

From the NVtop documentation: "NVTOP supports AMD GPUs using the amdgpu driver through the exposed DRM and sysfs interface.

AMD introduced the fdinfo interface in kernel 5.14 (browse kernel source). Hence, you will need a kernel with a version greater or equal to 5.14 to see the processes using AMD GPUs.

Support for recent GPUs are regularly mainlined into the linux kernel, so please use a recent-enough kernel for your GPU."

Additional information:

@nicolargo
Copy link
Owner Author

nicolargo commented Apr 2, 2024

Is anyone with a AMD GPU can copy paste the result of the command available in the Gist: https://gist.github.com/nicolargo/639fb23baaedf3c7ce29f4f9de88548b (please add a comment directly in the Gist) ?

cc: @PhilipDeegan

@HarlemSquirrel
Copy link

FYI I figured out what a lot of those do in this ruby gem: https://github.com/HarlemSquirrel/amdgpu-fan-rb/

nicolargo added a commit that referenced this issue Apr 7, 2024
@nicolargo
Copy link
Owner Author

nicolargo commented Apr 7, 2024

Implementation done and merged into develop.

Need test. For the moment only Linux OS with kernel >= 5.14 are supported.

cc: @PhilipDeegan @kdbanman @douglasg14b @hunasdf can you test the develop branch ?

@HarlemSquirrel
Copy link

HarlemSquirrel commented Apr 7, 2024

By needs test do you mean automated unit test or have someone manually test?

I can definitely manually test.

@HarlemSquirrel
Copy link

I just built from latest develop branch and ran with make run and I see some info!

image

@artem-zinnatullin
Copy link

Hm, I don't see AMD GPU 7900XTX running webserver via Docker from nicolargo/glances:ubuntu-dev 7f843c53663f image published 2 days ago.

Is it included there?

My kernel is Linux 6.5.0-27-generic.

For reference https://github.com/Umio-Yasuno/amdgpu_top successfully displays data from my 7900XTX, but nvtop doesn't Syllo/nvtop#236

@artem-zinnatullin
Copy link

Update: I don't know how I missed it in the Web UI, but it's definitely working in Docker (actually containerd + K8S) and even displays integrated GPU data!

image

This is great ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests