GPU and Node Utilization JSON API (CPU, GPU, Memory)

[altahat2003]

Description

This PR adds two utilization APIs:

• variorum_get_node_utilization_json: it populates a string in JSON format with total CPU node utilization, user CPU utilization, kernel CPU utilization, total node memory utilization, and GPU utilization.
• variorum_get_gpu_utilization_json: it populates string in JSON format with utilization of each GPU

Fixes #440

Docs: https://variorum.readthedocs.io/en/pr-from-fork-431/

Closes #495 once all tests pass.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature/architecture support (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Build/CI update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please provide hardware architecture specs and
instructions so we can reproduce.

• I rebuilt variorum on Lassen and test added API with a simple hello world example.

Checklist:

• I have run ./scripts/check-code-format.sh and confirm my code code follows the style guidelines of variorum
• I have added comments in my code
• My changes generate no new warnings (build with -DENABLE_WARNINGS=ON)
• New and existing unit tests pass with my changes

[tpatki]

Capturing discussion here as we'll need to resolve the comments that are on outdated commits.

@altahat2003

The getrusage part will be common to all architectures, and we will have to maintain CPU and memory delta values (save initial state). My suggestion is that you move this code out to a higher level in variorum.c itself instead of creating redundant code in each lower-level platform file. You don't really need a platform-specific implementation for the CPU and memory usage part. Just for the GPU part.

Only the GPU usage will be different based on which architecture you are on. We have a print_gpu_utilization API, and we can add a new get_gpu_utilization_json API, that you can then call from variorum.c.

Collaborator
@amarathe84 amarathe84 5 days ago
Summarizing the suggestion from our standup call here: Could the high-level utilization API be parameterized to choose between (a) reporting accumulator-style data when available on the underlying system or (b) reporting delta between current and previous sample? This will satisfy a variety of use cases depending on the monitoring requirements and configurations of the client application. For example, RoCM reports GPU utilization via. both accumulator-style utilization API as well as instantaneous utilization API. So the client could select which underlying API it wants to use.

[tpatki]

To do before merging:

• Rebase
• Add example for get_gpu_utilization_json
• Add docs for get_gpu_utilization_json
• Update description to show that this PR adds 2 APIs: get_gpu_utilization_json and get_node_utilization_json.