[core] Fix POSIX semaphore crash in experimental mutable objects (#62328)

[tobby168]

Why are these changes needed?

Closes #62328.

OpenSemaphores() in experimental_mutable_object_manager.cc crashes with

Check failed: RAY_PREDICT_TRUE(_left_ != _right_) 0 vs 0
  [src/ray/core_worker/experimental_mutable_object_manager.cc:177]

when a reader registers a channel in multi-node setups — e.g. vLLM compiled graphs running tensor parallelism across two DGX Spark nodes.

This PR addresses two independent bugs observed while debugging #62328.

1. Cross-node reader skips O_CREAT (primary bug)

PlasmaObjectHeader::semaphores_created is an atomic flag that sits inside the object's shared-memory header. It coordinates semaphore initialization within a single machine's shared memory, not across nodes.

On a different node, the reader's mapped header can observe kDone (propagated through the object header contents) even though the underlying named POSIX semaphores were never created on the reader's machine — named semaphores are machine-local. The reader then calls

sem_open(name, 0);   // no O_CREAT

which fails with ENOENT and returns SEM_FAILED. The subsequent RAY_CHECK_NE(sem, SEM_FAILED) fires and the worker aborts.

Fix: use O_CREAT (without O_EXCL) on the reader path. In the single-node case this just opens the existing semaphores; in the cross-node case it creates them locally with the correct initial value of 1, matching the writer path.

2. Non-POSIX semaphore names

POSIX specifies that sem_open() names must start with /. glibc is lenient and silently strips a missing slash, so this works today on most Linux distros. Stricter implementations (musl, some older libcs) reject such names with EINVAL.

Fix: change the prefixes from "obj"/"hdr" to "/obj"/"/hdr", and bound the resulting full name against PSEMNAMLEN (not just the raw name) so the 4-byte prefix can't push the result over macOS's 30-byte limit.

Related work

#62373 addresses the same issue with a very similar approach. I wrote this independently while investigating #62328 on a DGX Spark multi-node setup; the only notable deltas here are:

• Two regression tests (TestSemaphoreNameHasLeadingSlash, TestCrossNodeSemaphoreCreation) pinning both behaviors, including the cross-node kDone-without-semaphores scenario.
• Updated inline comments explaining the cross-node reasoning and the glibc-vs-other-libc semantics.

Happy to close this in favor of #62373 if @stephanie-wang prefers; flagging both options so whoever reviews can pick whichever they like.

Related issue number

Closes #62328
Related: #62373

Checks

• I've signed off every commit (by using the `-s` flag, i.e., `git commit -s`) in this PR.
• I've run `scripts/format.sh` to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • (No doc changes; bug fix only.)
• I've added any new APIs to the API Reference. If the API is intended to be private, it has been properly renamed.
  • (No new APIs.)
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

cc @stephanie-wang @jackhumphries

[gemini-code-assist]

Code Review

This pull request improves POSIX compliance and cross-node support for mutable object semaphores by prepending a leading slash to names and using O_CREAT in the reader path. Feedback suggests updating semaphore name length checks to account for the added prefix and wrapping platform-specific test code in conditional compilation for macOS compatibility.

[tobby168]

Thanks for the review! Addressed all three points in 4792c0f.

1. RAY_CHECK_LE on ret.size() instead of name.size() — you're right. With PSEMNAMLEN = 30 on macOS, the existing check only guarantees name.size() <= 30, so the "/obj" / "/hdr" prefix could push the final name to 34 bytes and fail with ENAMETOOLONG before sem_open even returns cleanly. Changed both helpers to bound ret.size().

   As a side note: this means my original PR body's claim that this version is safer than fix: semaphore open failures in MutableObjectManager cross-node setup #62373 on this dimension was exactly backwards. fix: semaphore open failures in MutableObjectManager cross-node setup #62373's RAY_CHECK_LE(name.size() + 4, PSEMNAMLEN) was already the correct bound; my RAY_CHECK_LE(name.size(), PSEMNAMLEN) was the looser (and buggier) one. Apologies for the miscall — corrected now by bounding ret.size() directly, which avoids the +4 magic number.

2. sem_getvalue on macOS — verified against Darwin XNU: sem_getvalue isn't implemented (returns -1 with errno=ENOSYS). Wrapped the initial-value probe in #ifdef __linux__. The RegisterChannel success assertion — which is the actual regression guard for [Bug] experimental_mutable_object_manager.cc - Check failed: RAY_PREDICT_TRUE(_left_ != _right_) 0 vs 0 #62328 — still runs on both platforms.

3. (Drive-by) Also corrected a stale "Linux enforces strictly via glibc" phrasing in the test comment that didn't match the accurate description in the .cc file.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 4792c0f. Configure here.}

[stephanie-wang]

Thanks for this fix!