Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
replace resource-hwloc module with enhanced flux-hwloc command #1968
The work here is not quite done, but I wanted to post a functional PR early since I'm sure there will be many comments...
This PR is an alternative approach to #1931. The resource-hwloc module itself is removed in favor of an enhanced
This approach removes the possible deadlocks introduced in #1931 since modules can be loaded/unloaded from any rank at any time. Doing the hwloc reload from a command interface was just much easier to reason about.
In this PR the
@@ Coverage Diff @@ ## master #1968 +/- ## ========================================== + Coverage 80.07% 80.19% +0.11% ========================================== Files 195 195 Lines 35048 35077 +29 ========================================== + Hits 28064 28129 +65 + Misses 6984 6948 -36
Rebased and pushed. Also added
I was able to do some testing at about 2K brokers on opal (across 32 real nodes), and sadly a
You can see below the difference between reloading local XML from all ranks vs just one
Also, I'm not sure if reloading XML from a directory via a single transaction in
For comparison, the timing for
Actually, not so cool. I might have not been clear in my benchmarks, but the preferred approach in this PR (replacing the resource-hwloc module with
Suppress uninitialized variable warnings from Valgrind under optparse_add_doc() by ensuring the entire structure is initialized instead of initializing individual fields.
Add libaggregate to src/cmd/Makefile.am in preparation for its requirement by flux-hwloc subcommand.
The --walk-topology flag for flux-hwloc reload is not used in any real world scenario. Remove it.
Add ability for flux-hwloc to load local topology using the same flags as used in the resource-hwloc module. Add a "--local" option to the `info` and `topology` subcommands to dump info or XML for the local system topology instead of the global topology.
Add support for fetching hwloc topology XML directly from kvs instead of the resource-hwloc.topo RPC. Add a `--rank` option to flux-hwloc topology and info subcommands to allow dumping a subset of topology information.
Convert cmd_lstopo() to use flux_hwloc_global_topology(). Drop code used to fetch hwloc topology with an rpc.
Add an undocument `aggregate-load` subcommand to flux-hwloc. This subcommand is meant to be run across all ranks to aggregate a JSON topology summary and optionally store it in the KVS. The subcommand also optionally loads local hwloc topology XML into the appropriate resource.hwloc.xml directory for the local rank.
This commit replaces the `reload` subcommand which uses the resource-hwloc.reload RPC with a version that directly loads hwloc topology XML to kvs. When loading XML files from a directory path, the data is directly put into the kvs under a single transaction from the `flux hwloc reload` command. For reloading system topology, the reload command executes the `flux hwloc aggregate-load` command to load XML on each rank. In either case, `flux hwloc reload` executes `aggregate-load` across all ranks to re-aggregate a topology summary JSON object, which is placed at `resource.hwloc.by_rank` instead of the broken down per-rank directories.
Some tests will now use jq for parsing and validating JSON output from commands, so include it in the 'testenv' docker images.