-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sample code to show how can hwloc be used to print latency /bandwidth for initiators and targets #542
Comments
Hello |
Closes open-mpi#542 Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
Would this example help https://github.com/bgoglin/hwloc/blob/master/doc/examples/memory-attributes.c ? It shows this on my laptop (no heterogeneous memory, no HMAT):
And this on a machine with 2 kinds of memory and HMAT:
|
Closes open-mpi#542 Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>
Hi Brice, First of all let me thank you for the quick reply and code . Excellent support , appreciated Please keep the issue open for a week or so to add feedback or if there is a different area for Q/A discussion we can use that. Again Thanks a bunch |
This issue will be autoclosed by github when I commit the new example to the repository, but feel free to ask other questions here, I'll be notified. |
Hi Does hwloc work in qemu enviroment setup with HMAT >? ls /sys/firmware/acpi/tables lstopo when i run the sample : There are 3 NUMA nodes |
Yes I use qemu quite intensively for simulating strange memory subsystems. But it doesn't populate HMAT with anything useful by default. There's an example of what I've been using in section "HMAT performance attributes" of https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu
|
Hi Brice, The XML file idea is brilliant . just 2 more questions : |
I don't have a XML with latency in the repo. But adding latencies isn't very hard:
|
Hi, Thanks for the Reply , much appreciated I tried to modify the XML file
when I changed the "Flag=6" I start seeing the Latency values :) . Thanks |
Ah sorry, I forgot that you need to change the "flags" after Bandwidth. For latency it's 6 instead of 5 (because best latency is lowest while best bandwidth is highest). |
Hi, I have 2 questions :
I like to know the penalty in terms of attributes for a CPU core to access memory outside of its local socket./package . (something like NUMA node local latency + latency to go off socket to access the memory )
|
@aghozzo For question 1, the mechanism already exists in hwloc, and in the ACPI HMAT table in hardware. Unfortunately the Linux kernel currently only exposes latency/bandwidth information for local CPUs (kernel developers were afraid of the matrices being too huge in memory on future machine :/). So hwloc can give you the information but it cannot get it from Linux. If you give the info to hwloc (either through XML or through the memattrs API), it'll be able to return it to you later. For question 2, it depends on what the ACPI tables expose. If they don't expose any locality information, the CPU-less node is attached to the root object at the top of the hwloc topology and appears local to all cores. If ACPI HMAT says the best initiator is CPU0, hwloc will attached the node to that CPU. In practice, HBM and NVDIMM have no local CPUs in hardware (because CPUs are rather local to DRAM) but modern platforms are supposed to explicit their local CPUs in the HMAT (hence hwloc will expose them as local to some CPUs). |
As always, thanks for the quick reply, I really appreciate it.
--> is there a reference for this , I like to study more on this subject.
|
About exposing only local CPU performance, I am trying to find the email in the linux kernel archives but I couldn't yet. It comes from when Ross Zwisler posted the ACPI HMAT patches in 2017 but there are multiple versions of the patchset and many replies. See below for more details. numactl --hardware is different, it's really a matrix of distances which comes from the ACPI SLIT table which exposes relative latencies between cores of one NUMA node and memory of another node. ACPI HMAT is a modern table that improves SLIT by defining "initiator" and "target" to support CPU-less and memory-less nodes better, adding bandwidth, etc. Both ACPI tables (when implemented by the platform) provide entire matrices with values between all pairs of nodes) but the difference is how the kernel exposes them. SLIT is exposed in /sys/devices/system/node/nodeX/distance, one single file per node with all distances to all other nodes. HMAT is exposed in a more complex manner because it defines the notion of best initiator, etc. Individual values are in /sys/devices/system/node/nodeX/accessY/initiators/{read,write}_{bandwidth,latency}. My understanding is that exposing all HMAT values in these hierarchy will create many sysfs subdirectories and files, that's why kernel developers didn't want to expose everything first. I guess they should keep this hierarchy for best initiator and just expose all raw values separately to avoid memory issues with sysfs files. In practice ACPI SLIT is always implemented, but sometimes it's useless because it just says 10 for local node and 20 for all remote nodes without distinguish close and far nodes. These values are exposed in hwloc "distances". ACPI HMAT is rather recent, basically only implemented since IceLake. Values seem to be more accurately defined by vendors so far. Those values are exposed in hwloc memattrs. SLIT distances and HMAT latencies should be similar in theory (except that SLIT values are normalized to 10 for local) but it's not guaranteed in practice. Also SLIT contains more values since it exposes latencies between pairs of memory-less nodes (even if they don't make sense since we need "CPUs" to define that latency). The LOCALITY attribute is defined in hwloc/memattrs.h as the number of PUs near a NUMA node, basically the weight of its cpuset. It's designed for platforms where different kinds of memory are attached at different level, for instance one NVM node per machine, one DRAM node per socket, and one HBM per half-socket (e.g. SNC). In this case the LOCALITY of NVM would be the number of PUs in the machine > LOCALITY of DRAM (PUs per socket) > LOCALITY of HBM (PUs per half socket). It's not clear it'll be useful in practice, but it was easy to implement. lstopo --distances -p would print the equivalent of numactl --hardware (but the ordering of nodes might be different since hwloc reorders node by logical index). It's implemented in https://github.com/open-mpi/hwloc/blob/master/utils/lstopo/lstopo-text.c#L218 which ends up calling https://github.com/open-mpi/hwloc/blob/master/utils/hwloc/misc.h#L382 |
@bgoglin Thanks a lot for the detailed reply , appreciated Does hwloc support other allocation calls other than malloc ? ( I mean calloc, realloc, memalign ) . I use "hwloc_alloc_membind()" to do allocations on best target from a specific initiator . is there an API for other calls ? or I can set the policy through "hwloc_set_membind" and then do normal malloc/calloc/realloc and that will allocate to best_node i set in the "hwloc_set_membind" ? |
Hello
|
I couldn't find any sample C code that shows how to extract HMAT table attributes or APIs that can be used and called directly from a hello world C code .
can someone please provide a pointer on how to run and test sample code to get this information or if there are already exisiting examples ?
Thanks
The text was updated successfully, but these errors were encountered: