linux/sysfsnode: keep NVIDIA GPU NUMA nodes if HWLOC_KEEP_NVIDIA_GPU_…

…NUMA_NODES=1 in the environment

Use the PCI locality of the GPU to fix the NUMA locality (sysfsnode gives the entire machine cpuset),
hence it's attached as a second/third/... NUMA node to its local socket on P9.

The nodes' subtype is "GPUMemory".

They have a "PCIBusID" info attribute to identify the corresponding GPU.

Signed-off-by: Brice Goglin <Brice.Goglin@inria.fr>

NEWS

@@ -40,6 +40,8 @@ Version 2.1.0
   + The x86 now properly handles offline CPUs.
   + Detect the locality of NVIDIA GPU OpenCL devices.
   + Ignore NUMA nodes that correspond to NVIDIA GPU by default.
+    - They may be kept if HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES=1 in the environment.
+      Fix their CPU locality and add info attributes to identify them.
     Thanks to Max Katz and Edgar Leon for the help.
   + Expose Linux DAX devices as hwloc Block OS devices.
   + Add support for IBM S/390 drawers.

doc/hwloc.doxy

@@ -1050,6 +1050,16 @@ following environment variables.
   PCI locality reported by the platform is used.
   </dd>
 
+<dt>HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES=0</dt>
+  <dd>show or hide NUMA nodes that correspond to NVIDIA GPU memory.
+  By default they are ignored to avoid interleaved memory being allocated
+  on GPU by mistake.
+  Setting this environment variable to 1 exposes these NUMA nodes.
+  They may be recognized by the <em>GPUMemory</em> subtype.
+  They also have a <em>PCIBusID</em> info attribute to identify the
+  corresponding GPU.
+  </dd>
+
 <dt>HWLOC_FSROOT=/path/to/linux/filesystem-root/</dt>
   <dd>switches to reading the topology from the specified
   Linux filesystem root instead of the main file-system root.
@@ -1635,6 +1645,8 @@ It may also specify that a Block OS device is a <em>Disk</em>,
 or that a CoProcessor OS device is a <em>CUDA</em> device.
 This subtype is displayed by lstopo either in place or after the
 main <tt>obj->type</tt> attribute.
+NUMA nodes that correspond GPU memory may also have <em>GPUMemory</em>
+as subtype.
 
 Each object also contains an <tt>attr</tt> field that, if non NULL,
 points to a union ::hwloc_obj_attr_u of type-specific attribute
@@ -1863,6 +1875,9 @@ These info attributes are attached to objects specified in parentheses.
 <dd>The name of the Linux DAX device that was used to expose a non-volatile
 memory region as a volatile NUMA node.
 </dd>
+<dt>PCIBusID (GPUMemory NUMA Nodes)</dt>
+<dd>The PCI bus ID of the GPU whose memory is exposed in this NUMA node.
+</dd>
 <dt>Inclusive (Caches)</dt>
 <dd>The inclusiveness of a cache (1 if inclusive, 0 otherwise).
 Currently only available on x86 processors.

hwloc/topology-linux.c

@@ -3863,6 +3863,8 @@ look_sysfsnode(struct hwloc_topology *topology,
       dir = hwloc_opendir("/proc/driver/nvidia/gpus", data->root_fd);
       if (dir) {
 	struct dirent *dirent;
+	char *env = getenv("HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES");
+	int keep = env && atoi(env);
 	while ((dirent = readdir(dir)) != NULL) {
 	  char nvgpunumapath[300], line[256];
 	  int fd;
@@ -3884,9 +3886,22 @@ look_sysfsnode(struct hwloc_topology *topology,
 		for(i=0; i<nbnodes; i++) {
 		  hwloc_obj_t node = nodes[i];
 		  if (node && node->os_index == nvgpu_node) {
-		    /* drop this NUMA node */
-		    hwloc_free_unlinked_object(node);
-		    nodes[i] = NULL;
+		    if (keep) {
+		      /* keep this NUMA node but fixed its locality and add an info about the GPU */
+		      char nvgpulocalcpuspath[300];
+		      int err;
+		      node->subtype = strdup("GPUMemory");
+		      hwloc_obj_add_info(node, "PCIBusID", dirent->d_name);
+		      snprintf(nvgpulocalcpuspath, sizeof(nvgpulocalcpuspath), "/sys/bus/pci/devices/%s/local_cpus", dirent->d_name);
+		      err = hwloc__read_path_as_cpumask(nvgpulocalcpuspath, node->cpuset, data->root_fd);
+		      if (err)
+			/* the core will attach to the root */
+			hwloc_bitmap_zero(node->cpuset);
+		    } else {
+		      /* drop this NUMA node */
+		      hwloc_free_unlinked_object(node);
+		      nodes[i] = NULL;
+		    }
 		    break;
 		  }
 		}