feat: support Gather with OpenMPI backend implementation

[GordonYang1]

Summary

This PR adds collective Gather support to InfiniCCL with an OpenMPI-based backend implementation, along with an example program for functional verification and basic performance evaluation.

Gather collects one equally-sized block from every rank onto a single root rank. The public API is exposed through infinicclGather() with the signature (sendbuff, recvbuff, count, datatype, root, comm, stream), consistent with the existing rooted collectives. The current implementation host-stages device buffers and delegates to a blocking MPI_Gather internally.

Changes

• Public Gather API

  • add the public API declaration for infinicclGather() in include/comm.h;
  • the collective is exposed through the common communicator interface; the extern "C" entry is produced by the existing auto-generated bridge, so no manual wiring is needed.

• Base Gather Wrapper

  • add src/base/gather.h;
  • validate the communicator handle, datatype value, root rank range, and buffer pointers (the receive buffer is only required on root) before dispatching to the backend implementation;
  • treat a zero count as a successful no-op;
  • return infinicclInvalidArgument for invalid user inputs.

• OpenMPI-based Gather Implementation

  • add src/ompi/impl/gather.h;
  • implement the collective with a blocking MPI_Gather, allocating the host receive buffer only on root, where the gathered result lands;
  • use temporary host-staging buffers for device-memory transfer (device-to-host before, host-to-device on root after);
  • map DataType to the corresponding MPI_Datatype, and reject a count exceeding INT_MAX.

• Gather Example

  • add examples/gather.cc;
  • align the example structure and output style with the existing example programs such as all_reduce, all_gather, reduce_scatter, broadcast, and all_to_all;
  • each rank contributes (rank + 1), and the example verifies that root receives the [1, 2, …, size] blocks in rank order;
  • report basic timing and bandwidth metrics in the same style as the other examples.

Platform and Backend Affected

Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU

Backend

• OpenMPI
• MPICH

Performance Impact

• No performance impact
• Performance improved
• Performance regression possible

This PR adds a new collective and does not change the performance of existing operations. For reference, the heterogeneous run (16 ranks, 4 MB per rank) measured Gather at 42.343 ms, 2.77 GB/s bus bandwidth, 1.48 GB/s algorithm bandwidth.

Known Issues & Future Work

• The current OpenMPI Gather implementation is blocking and does not overlap communication with computation. Future work may introduce stream-aware asynchronous execution.
• The current implementation allocates temporary host-staging buffers using malloc/free on every invocation. Future work may add reusable buffer pools, allocator caching, and pinned host memory to reduce per-call overhead.
• The current implementation rejects a count larger than INT_MAX instead of splitting it into INT_MAX-bounded chunks (as the point-to-point path does). Future work may add chunking for very large transfers.
• The current implementation performs GPU-to-Host and Host-to-GPU copies. While functionally correct, this is not optimal for GPU-intensive workloads. Future work may implement zero-copy GPU-GPU transfers or native GPU collectives (e.g. NCCL / CNCL) where supported.
• The current example intentionally follows the lightweight style of the existing example programs. Future dedicated tests may add broader coverage such as a non-zero root, zero-count calls, mismatched-buffer validation, and large-count stress cases.

Test Results

Validated on a MetaX–NVIDIA heterogeneous cluster over the OpenMPI backend via scripts/run_examples.py:

• server: NVIDIA, 8 GPUs, ranks 0–7 (built with Devices [cpu, nvidia], Backends [ompi]).
• test: MetaX, 8 GPUs, ranks 8–15 (built with Devices [cpu, metax], Backends [ompi]).
• 16 ranks total; message size 1,048,576 float32 (4 MB) per rank; 2 warm-up + 20 profiled iterations.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU

Test Involved Backend

• OpenMPI
• MPICH

all_gather.log
all_reduce.log
all_to_all.log
broadcast.log
gather.log
reduce.log
reduce_scatter.log
send_recv.log

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from master — the branch is rebased cleanly on top of the current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• N/A- Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

• N/A- Code is PEP 8 compliant; ruff check passes cleanly on CI (see .github/workflows/ruff.yml).
• N/A- ruff format --check passes cleanly — if not, run ruff format and commit the result.
• N/A- Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• N/A- Framework-specific conventions (e.g. lowercase pytest.skip messages without terminal period) are honored where applicable (CONTRIBUTING.md §Python).
• N/A- No blank line between the function signature and the body when there is no docstring or comment (CONTRIBUTING.md §Python).
• N/A- A blank line is present before and after if, for, and similar control-flow statements (CONTRIBUTING.md §Python).
• N/A- A blank line appears before each return, except when it directly follows a control-flow statement (CONTRIBUTING.md §Python).
• N/A- Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• N/A- Type hints are added / kept consistent with the surrounding code.

Testing

• All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

• N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
• Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

• N/A- README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A- Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• N/A- Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3b0d6cc1bd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Ziminli]

PR 模板麻烦更新到最新的模板。