Use hybrid MR mode with the cxi provider

[jhh67]

As described by @gbtitus:

The libfabric cxi provider for Cassini-based networks now supports the
so-called "hybrid" local memory registration mode. For functions that
accept a local MR descriptor, this allows the client to pass the
non-NULL MR descriptor (pointer) if the local address is already
registered, or NULL to indicate that the local memory is not registered
and the provider should use its internal MR cache. (Absent this hybrid
mode the provider always uses its internal MR cache, which is a source
of contention for multi-threaded clients such as Chapel and in certain
cases SHMEM.)

Resolves Cray/chapel-private#2957.

Signed-off-by: John H. Hartman jhh67@users.noreply.github.com

[jhh67]

Regarding rdma/fi_cxi_ext.h, it can't/won't be included in a core libfabric include file so we'll have to conditionally include it ourselves.

I confirmed that fi_open_ops returns "Invalid argument" (-22) if called with an invalid name argument, so that we will correctly fall back to non-hybrid MR mode if using a cxi provider that doesn't support it. I tested this by compiling against a version of rdma/fi_cxi_ext.h that defined FI_CXI_DOM_OPS_3 and linking against a version of libfabric that did not implement this option.

[ronawho]

Looks good!

Full -multilocale-only testing was clean. See https://github.com/Cray/chapel-private/issues/3153 for instructions on running testing going forward.

Performance also looks good. This significantly improves thread scaling for fetching atomics and GETs. Here's results for the comm-ops microbenchmark:

Fetching AMO:

cores	main	hybrid
1	1.55s (0.32 Mops/s)	1.47s ( 0.34 Mops/s)
4	2.32s (0.86 Mops/s)	1.50s ( 1.33 Mops/s)
16	8.56s (0.93 Mops/s)	1.53s ( 5.23 Mops/s)
64	45.73s (0.70 Mops/s)	1.60s (19.98 Mops/s)

GET:

cores	main	hybrid
1	1.47s (0.34 Mops/s)	1.42s ( 0.35 Mops/s)
4	2.28s (0.88 Mops/s)	1.47s ( 1.36 Mops/s)
16	8.43s (0.95 Mops/s)	1.50s ( 5.35 Mops/s)
64	44.79s (0.71 Mops/s)	1.56s (20.54 Mops/s)

And here's results for 16-node fine-grained indexgather:

cores	main	hybrid
1	0.008 GB/s/node	0.008 GB/s/node
4	0.014 GB/s/node	0.018 GB/s/node
16	0.016 GB/s/node	0.071 GB/s/node
64	0.011 GB/s/node	0.236 GB/s/node