Merge branch 'master' into distrib

.github/contributing.md

@@ -0,0 +1,31 @@
+# License and Signed-off-by token
+
+In order to ensure that we can keep distributing hwloc under our
+[open source license](/COPYING), we need to ensure that all
+contributions are compatible with that license.
+
+To that end, we require that all Git commits contributed to hwloc
+have a "Signed-off-by" token indicating that the commit author agrees
+with [Open MPI's Contributor's
+Declaration](https://github.com/open-mpi/ompi/wiki/Administrative-rules#contributors-declaration).
+
+If you have not already done so, please ensure that:
+
+1. Every commit contains exactly the "Signed-off-by" token.  You can
+add this token via `git commit -s`.
+1. The email address after "Signed-off-by" must match the Git commit
+email address.
+
+# Copyright
+
+You may also update the copyright headers whenever you modify
+a file. `contrib/update-my-copyright.pl` may help you doing so.
+It requires you to set the `HWLOC_COPYRIGHT_FORMAT_NAME` environment
+to something like `Inria.  All rights reserved.`
+and `HWLOC_COPYRIGHT_SEARCH_NAME` to something like `Inria`.
+
+You may even call `contrib/update-my-copyright.pl --check-only` from
+the git pre-commit hook so that it prevents committing without
+updated copyright headers (unless `--no-verify` is given).
+
+Major contributors are also listed in the [Authors](/AUTHORS) file.

.github/issue_template.md

@@ -27,9 +27,8 @@
 If your issue consists in a wrong topology detection, we also need the following for debugging remotely:
 
 * On Linux, run `hwloc-gather-topology myhost` and post the `myhost.*` files that it will generate. Note that this tool may be slow on large nodes or when I/O is enabled.
-* On MacOS, `sysctl hw` and `sysctl machdep.cpu`
-* On Solaris, `kstat cpu_info` and `lgrp_info -a`
-* On BSD, `sysctl hw`
+* On Solaris, `kstat -C cpu_info`, `lgrpinfo -a` and `psrinfo`
+* On MacOS and BSD, `sysctl hw` and `sysctl machdep`
 * On BSD x86 platforms, if using hwloc >= 2.0, also run `hwloc-gather-cpuid` and post an archive of the `cpuid` that was generated.
 * On Windows, `coreinfo -cgnlsm`
 
@@ -38,3 +37,4 @@ If your issue consists in a wrong topology detection, we also need the following
 You may need to archive the output from the above commands into a `.zip` or `.tar.gz` (not `.bz2`!) file before Github will allow you to drag-n-drop the file into the issue to attach it.
 
 Note that upgrading your operating system (e.g., Linux kernel) and platform firmwares (e.g., BIOS) might help solving issues about wrong topology detection.
+Some known issues are listed at https://github.com/open-mpi/hwloc/wiki/Linux-kernel-bugs

.gitignore

@@ -112,7 +112,6 @@ test-suite.log
 /tests/hwloc/hwloc_iodevs
 /tests/hwloc/xmlbuffer
 /tests/hwloc/gl
-/tests/hwloc/intel-mic
 /tests/hwloc/linux-libnuma
 /tests/hwloc/glibc-sched
 /tests/hwloc/openfabrics-verbs
@@ -130,6 +129,8 @@ test-suite.log
 
 /tests/hwloc/x86/test-topology.sh
 
+/tests/hwloc/x86+linux/test-topology.sh
+
 /tests/hwloc/xml/test-topology.sh
 
 /tests/hwloc/embedded/aclocal.m4
@@ -197,6 +198,7 @@ test-suite.log
 /utils/lstopo/lstopo-no-graphics
 /utils/lstopo/lstopo-no-graphics.1
 /utils/lstopo/test-lstopo.sh
+/utils/lstopo/test-lstopo-shmem.sh
 /utils/lstopo/test-*.sh.log
 /utils/lstopo/test-*.sh.trs
 

NEWS

@@ -16,9 +16,16 @@ bug fixes (and other actions) for each version of hwloc since version
 0.9.
 
 
+Version 2.2.0
+-------------
+* Remove support for Intel Xeon Phi (MIC, Knights Corner) co-processors.
+
+
 Version 2.1.0
 -------------
 * API
+  + Add a new "Die" object (HWLOC_OBJ_DIE) for upcoming x86 processors
+    with multiple dies per package, in the x86 and Linux backends.
   + Add the new HWLOC_OBJ_MEMCACHE object type for memory-side caches.
     - They are filtered-out by default, except in command-line tools.
     - They are only available on very recent platforms running Linux 5.2+
@@ -34,22 +41,32 @@ Version 2.1.0
   + Improve the API for dealing with disallowed resources
     - HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM is replaced with FLAG_INCLUDE_DISALLOWED
       and --whole-system command-line options with --disallowed.
-      Former names are still accepted for backward compatibility.
+      . Former names are still accepted for backward compatibility.
     - Add hwloc_topology_allow() for changing allowed sets after load().
     - Add the HWLOC_ALLOW=all environment variable to totally ignore
       administrative restrictions such as Linux Cgroups.
     - Add disallowed_pu and disallowed_numa bits to the discovery support
       structure.
+  + Group objects have a new "dont_merge" attribute to prevent them from
+    being automatically merged with identical parent or children.
+  + Add more distances-related features:
+    - Add hwloc_distances_get_name() to retrieve a string describing
+      what a distances structure contain.
+    - Add hwloc_distances_get_by_name() to retrieve distances structures
+      based on their name.
+    - Add hwloc_distances_release_remove()
+    - Distances may now cover objects of different types with new kind
+      HWLOC_DISTANCES_KIND_HETEROGENEOUS_TYPES.
 * Backends
+  + Add support for Linux 5.3 new sysfs cpu topology files with Die information.
   + Add support for Intel v2 Extended Topology Enumeration in the x86 backend.
-    Tiles, Modules and Dies are exposed as Groups for now.
   + Improve memory locality on Linux by using HMAT initiators (exposed
     since Linux 5.2+), and NUMA distances for CPU-less NUMA nodes.
-  + The x86 now properly handles offline CPUs.
+  + The x86 backend now properly handles offline CPUs.
   + Detect the locality of NVIDIA GPU OpenCL devices.
   + Ignore NUMA nodes that correspond to NVIDIA GPU by default.
-    - They may be kept if HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES=1 in the environment.
-      Fix their CPU locality and add info attributes to identify them.
+    - They may be unignored if HWLOC_KEEP_NVIDIA_GPU_NUMA_NODES=1 in the environment.
+    - Fix their CPU locality and add info attributes to identify them.
     Thanks to Max Katz and Edgar Leon for the help.
   + Add support for IBM S/390 drawers.
   + Rework the heuristics for discovering KNL Cluster and Memory modes
@@ -60,12 +77,17 @@ Version 2.1.0
   + Expose Linux DAX devices as hwloc Block OS devices.
   + Remove support for /proc/cpuinfo-only topology discovery in Linux
     kernel prior to 2.6.16.
+  + Disable POWER device-tree-based topology on Linux by default.
+    - It may be reenabled by setting HWLOC_USE_DT=1 in the environment.
+  + Discovery components are now divided in phases that may be individually
+    blacklisted.
+    - The linuxio component has been merged back into the linux component.
 * Tools
   + lstopo
     - lstopo factorizes objects by default in the graphical output when
       there are more than 4 identical children.
-      New options --no-factorize and --factorize may be used to configure this.
-      Also hit the 'f' key to disable factorizing in interactive outputs.
+      . New options --no-factorize and --factorize may be used to configure this.
+      . Hit the 'f' key to disable factorizing in interactive outputs.
     - Both logical and OS/physical indexes are now displayed by default
       for PU and NUMA nodes.
     - The X11 and Windows interactive outputs support many keyboard
@@ -78,9 +100,27 @@ Version 2.1.0
       See dynamic_SVG_example.html for an example.
   + Add --nodeset options to hwloc-calc for converting between cpusets and
     nodesets.
+  + Add --no-smt to lstopo, hwloc-bind and hwloc-calc to ignore multiple
+    PU in SMT cores.
   + hwloc-annotate may annotate multiple locations at once.
   + Add a HTML/JS version of hwloc-ps. See contrib/hwloc-ps.www/README.
   + Add bash completions.
+* Misc
+  + Add several FAQ entries in "Compatibility between hwloc versions"
+    about API version, ABI, XML, Synthetic strings, and shmem topologies.
+
+
+Version 2.0.4 (also included in 1.11.13 when appropriate)
+-------------
+* Add support for Linux 5.3 new sysfs cpu topology files with Die information.
+* Add support for Intel v2 Extended Topology Enumeration in the x86 backend.
+* Tiles, Modules and Dies are exposed as Groups for now.
+  + HWLOC_DONT_MERGE_DIE_GROUPS=1 may be set in the environment to prevent
+    Die groups from being automatically merged with identical parent or children.
+* Ignore NUMA node information from AMD topoext in the x86 backend,
+  unless HWLOC_X86_TOPOEXT_NUMANODES=1 is set in the environment.
+* Group objects have a new "dont_merge" attribute to prevent them from
+  being automatically merged with identical parent or children.
 
 
 Version 2.0.3 (also included in 1.11.12 when appropriate)
@@ -267,6 +307,19 @@ Version 2.0.0
   + hwloc now requires a C99 compliant compiler.
 
 
+Version 1.11.13 (also included in 2.0.4)
+---------------
+* Add support for Linux 5.3 new sysfs cpu topology files with Die information.
+* Add support for Intel v2 Extended Topology Enumeration in the x86 backend.
+* Tiles, Modules and Dies are exposed as Groups for now.
+  + HWLOC_DONT_MERGE_DIE_GROUPS=1 may be set in the environment to prevent
+    Die groups from being automatically merged with identical parent or children.
+* Ignore NUMA node information from AMD topoext in the x86 backend,
+  unless HWLOC_X86_TOPOEXT_NUMANODES=1 is set in the environment.
+* Group objects have a new "dont_merge" attribute to prevent them from
+  being automatically merged with identical parent or children.
+
+
 Version 1.11.12 (also included in 2.0.3)
 ---------------
 * Fix a corner case of hwloc_topology_restrict() where children would

VERSION

@@ -8,7 +8,7 @@
 # Please update HWLOC_VERSION* in contrib/windows/hwloc_config.h too.
 
 major=2
-minor=1
+minor=2
 release=0
 
 # greek is used for alpha or beta release tags.  If it is non-empty,

config/hwloc.m4

@@ -1,6 +1,6 @@
 dnl -*- Autoconf -*-
 dnl
-dnl Copyright © 2009-2018 Inria.  All rights reserved.
+dnl Copyright © 2009-2019 Inria.  All rights reserved.
 dnl Copyright © 2009-2012, 2015-2017 Université Bordeaux
 dnl Copyright © 2004-2005 The Trustees of Indiana University and Indiana
 dnl                         University Research and Technology
@@ -217,8 +217,7 @@ EOF])
         AC_MSG_RESULT([Linux])
         hwloc_components="$hwloc_components linux"
         if test "x$enable_io" != xno; then
-	  hwloc_components="$hwloc_components linuxio"
-	  AC_DEFINE(HWLOC_HAVE_LINUXIO, 1, [Define to 1 if building the Linux I/O component])
+	  AC_DEFINE(HWLOC_HAVE_LINUXIO, 1, [Define to 1 for I/O discovery in the Linux component])
 	  hwloc_linuxio_happy=yes
 	  if test x$enable_pci != xno; then
 	    AC_DEFINE(HWLOC_HAVE_LINUXPCI, 1, [Define to 1 if enabling Linux-specific PCI discovery in the Linux I/O component])
@@ -1220,11 +1219,13 @@ return clGetDeviceIDs(0, 0, 0, NULL, NULL);
     AS_IF([test "$hwloc_mode" = "embedded"],
           [HWLOC_EMBEDDED_CFLAGS=$HWLOC_CFLAGS
            HWLOC_EMBEDDED_CPPFLAGS=$HWLOC_CPPFLAGS
+           HWLOC_EMBEDDED_LDFLAGS=$HWLOC_LDFLAGS
            HWLOC_EMBEDDED_LDADD='$(HWLOC_top_builddir)/hwloc/libhwloc_embedded.la'
            HWLOC_EMBEDDED_LIBS=$HWLOC_LIBS
            HWLOC_LIBS=])
     AC_SUBST(HWLOC_EMBEDDED_CFLAGS)
     AC_SUBST(HWLOC_EMBEDDED_CPPFLAGS)
+    AC_SUBST(HWLOC_EMBEDDED_LDFLAGS)
     AC_SUBST(HWLOC_EMBEDDED_LDADD)
     AC_SUBST(HWLOC_EMBEDDED_LIBS)
 

config/hwloc_internal.m4

@@ -438,13 +438,15 @@ int foo(void) {
         hwloc_config_prefix[tests/hwloc/linux/allowed/Makefile]
         hwloc_config_prefix[tests/hwloc/linux/gather/Makefile]
         hwloc_config_prefix[tests/hwloc/x86/Makefile]
+        hwloc_config_prefix[tests/hwloc/x86+linux/Makefile]
         hwloc_config_prefix[tests/hwloc/xml/Makefile]
         hwloc_config_prefix[tests/hwloc/ports/Makefile]
         hwloc_config_prefix[tests/hwloc/rename/Makefile]
         hwloc_config_prefix[tests/hwloc/linux/allowed/test-topology.sh]
         hwloc_config_prefix[tests/hwloc/linux/gather/test-gather-topology.sh]
         hwloc_config_prefix[tests/hwloc/linux/test-topology.sh]
         hwloc_config_prefix[tests/hwloc/x86/test-topology.sh]
+        hwloc_config_prefix[tests/hwloc/x86+linux/test-topology.sh]
         hwloc_config_prefix[tests/hwloc/xml/test-topology.sh]
         hwloc_config_prefix[tests/hwloc/wrapper.sh]
         hwloc_config_prefix[utils/hwloc/hwloc-compress-dir]
@@ -459,6 +461,7 @@ int foo(void) {
         hwloc_config_prefix[utils/hwloc/test-hwloc-dump-hwdata/Makefile]
         hwloc_config_prefix[utils/hwloc/test-hwloc-dump-hwdata/test-hwloc-dump-hwdata.sh]
         hwloc_config_prefix[utils/lstopo/test-lstopo.sh]
+        hwloc_config_prefix[utils/lstopo/test-lstopo-shmem.sh]
         hwloc_config_prefix[utils/netloc/infiniband/netloc_ib_gather_raw]
         hwloc_config_prefix[contrib/hwloc-ps.www/Makefile]
         hwloc_config_prefix[contrib/systemd/Makefile]
@@ -469,27 +472,29 @@ int foo(void) {
         hwloc_config_prefix[tests/netloc/tests.sh]
     )
 
-    AC_CONFIG_COMMANDS([chmoding-scripts], [
-chmod +x ]hwloc_config_prefix[tests/hwloc/linux/test-topology.sh \
-      ]hwloc_config_prefix[tests/hwloc/x86/test-topology.sh \
-      ]hwloc_config_prefix[tests/hwloc/xml/test-topology.sh \
-      ]hwloc_config_prefix[tests/hwloc/linux/allowed/test-topology.sh \
-      ]hwloc_config_prefix[tests/hwloc/linux/gather/test-gather-topology.sh \
-      ]hwloc_config_prefix[tests/hwloc/wrapper.sh \
-      ]hwloc_config_prefix[utils/hwloc/hwloc-compress-dir \
-      ]hwloc_config_prefix[utils/hwloc/hwloc-gather-topology \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-annotate.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-calc.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-compress-dir.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-diffpatch.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-distrib.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-info.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-fake-plugin.sh \
-      ]hwloc_config_prefix[utils/hwloc/test-hwloc-dump-hwdata/test-hwloc-dump-hwdata.sh \
-      ]hwloc_config_prefix[utils/lstopo/test-lstopo.sh \
-      ]hwloc_config_prefix[utils/netloc/infiniband/netloc_ib_gather_raw \
-      ]hwloc_config_prefix[contrib/windows/test-windows-version.sh \
-      ]hwloc_config_prefix[tests/netloc/tests.sh])
+    AC_CONFIG_COMMANDS([chmoding-scripts], [chmod +x] \
+      hwloc_config_prefix[tests/hwloc/linux/test-topology.sh] \
+      hwloc_config_prefix[tests/hwloc/x86/test-topology.sh] \
+      hwloc_config_prefix[tests/hwloc/x86+linux/test-topology.sh] \
+      hwloc_config_prefix[tests/hwloc/xml/test-topology.sh] \
+      hwloc_config_prefix[tests/hwloc/linux/allowed/test-topology.sh] \
+      hwloc_config_prefix[tests/hwloc/linux/gather/test-gather-topology.sh] \
+      hwloc_config_prefix[tests/hwloc/wrapper.sh] \
+      hwloc_config_prefix[utils/hwloc/hwloc-compress-dir] \
+      hwloc_config_prefix[utils/hwloc/hwloc-gather-topology] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-annotate.sh] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-calc.sh] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-compress-dir.sh] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-diffpatch.sh] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-distrib.sh] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-info.sh] \
+      hwloc_config_prefix[utils/hwloc/test-fake-plugin.sh] \
+      hwloc_config_prefix[utils/hwloc/test-hwloc-dump-hwdata/test-hwloc-dump-hwdata.sh] \
+      hwloc_config_prefix[utils/lstopo/test-lstopo.sh] \
+      hwloc_config_prefix[utils/lstopo/test-lstopo-shmem.sh] \
+      hwloc_config_prefix[utils/netloc/infiniband/netloc_ib_gather_raw] \
+      hwloc_config_prefix[contrib/windows/test-windows-version.sh] \
+      hwloc_config_prefix[tests/netloc/tests.sh])
 
     # These links are only needed in standalone mode.  It would
     # be nice to m4 foreach this somehow, but whenever I tried

contrib/ci.inria.fr/Jenkinsfile-basic

@@ -26,7 +26,7 @@ pipeline {
 				node('autotools') {
 					checkout scm
 					script {
-						gitBranch = sh (script: 'if test "$BRANCH_NAME"; then echo $BRANCH_NAME; else git branch | cut -c3-; fi', returnStdout: true).trim()
+						gitBranch = sh (script: 'if test "$BRANCH_NAME"; then echo $BRANCH_NAME; else git rev-parse --abbrev-ref HEAD; fi', returnStdout: true).trim()
 					}
 					sh 'contrib/ci.inria.fr/job-0-tarball.sh '+gitBranch
 					script {

contrib/ci.inria.fr/Jenkinsfile-extended

@@ -6,7 +6,7 @@
 //   + running Unix Debug checks
 //   + building MinGW Windows zipballs
 //   + running Embedded checks
-//   + running Sonarscanner analysis
+//   + running Coverity and Sonarscanner analysis
 //
 // The GIT checkout must use the remote branch name as the checkout local
 // branch name so that tarball names contain the branch name.
@@ -36,7 +36,7 @@ pipeline {
                                         checkout scm
 					script {
 						gitRepoURL = sh (script: 'git config --get remote.origin.url', returnStdout: true).trim()
-						gitBranch = sh (script: 'if test "$BRANCH_NAME"; then echo $BRANCH_NAME; else git branch | cut -c3-; fi', returnStdout: true).trim()
+						gitBranch = sh (script: 'if test "$BRANCH_NAME"; then echo $BRANCH_NAME; else git rev-parse --abbrev-ref HEAD; fi', returnStdout: true).trim()
 					}
                                         sh 'contrib/ci.inria.fr/job-0-tarball.sh '+gitBranch
                                         script {
@@ -47,6 +47,7 @@ pipeline {
                                         dir('contrib/ci.inria.fr') {
 						stash includes: "job-3-debug.sh", name: 'script-unix-debug'
 						stash includes: "job-3-embedded.sh", name: 'script-embedded'
+						stash includes: "job-3-coverity.sh", name: 'script-coverity'
 						stash includes: "job-3-sonarscanner.sh", name: 'script-sonarscanner'
 						stash includes: "job-3-mingw.*", name: 'scripts-mingw'
 						stash includes: "job-3-cygwin.*", name: 'scripts-cygwin'
@@ -70,7 +71,10 @@ pipeline {
 					if (env.NO_MINGW != 'true') {
 						listOfNodeNames.push('MinGW')
 					}
-					if (env.NO_SONAR != 'true') {
+					if (env.NO_COVERITY != 'true') {
+						listOfNodeNames.push('Coverity')
+					}
+					if (env.NO_SONARQUBE != 'true') {
 						listOfNodeNames.push('SonarQube Scanner')
 					}
 					if (env.NO_EMBEDDED != 'true') {
@@ -103,6 +107,16 @@ pipeline {
 										deleteDir()
 								}
 							}
+						} else if (it == 'Coverity') {
+							node('coverity') {
+								dir('check-coverity') {
+									unstash 'tarballgz'
+									unstash 'script-coverity'
+									sh 'chmod 755 job-3-coverity.sh && ./job-3-coverity.sh '+gitRepoURL+' '+gitBranch+' '+tarballgz
+									if (env.KEEP_WORKING_DIRECTORY != 'true')
+										deleteDir()
+								}
+							}
 						} else if (it == 'SonarQube Scanner') {
 							node('sonarscanner') {
 								dir('check-sonarscanner') {

contrib/ci.inria.fr/job-1-visualstudio.bat

@@ -1,5 +1,5 @@
 REM
-REM  Copyright © 2012-2018 Inria.  All rights reserved.
+REM  Copyright © 2012-2019 Inria.  All rights reserved.
 REM  See COPYING in top-level directory.
 REM
 
@@ -13,7 +13,7 @@ if %errorlevel% neq 0 exit /b %errorlevel%
 cd %TARBALL:~0,-7%\contrib\windows
 if %errorlevel% neq 0 exit /b %errorlevel%
 
-%MSBUILD_PATH%\MSBuild hwloc.sln /p:Configuration=Release /p:Platform=x64
+%MSBUILD_PATH%\MSBuild hwloc.sln /p:Configuration=Release /p:Platform=x64 /p:PlatformToolset=v110
 if %errorlevel% neq 0 exit /b %errorlevel%
 
 x64\Release\lstopo-no-graphics.exe