fix(heavier_once_cell): assertion failure can be hit

[koivunej]

@problame noticed that the tokio::sync::AcquireError branch assertion can be hit like in the first commit. We haven't seen this yet in production, but I'd prefer not to see it there. There take_and_deinit is being used, but this race must be quite timing sensitive.

[github-actions]

2394 tests run: 2277 passed, 0 failed, 117 skipped (full report)

---

Code coverage (full report)

• functions: 54.3% (11325 of 20843 functions)
• lines: 81.3% (63541 of 78122 lines)

---

_{The comment gets automatically updated with the latest test results
f5b0e72 at 2024-02-07T07:35:08.193Z :recycle:}

[koivunej]

Earlier you mentioned that the loser making any forward progress requires that there cannot be a forever loop which does get*_or_init + get_mut + take_and_deinit -- I agree. What would save us from that is to "take the queue" of the old semaphore and requeue them in the same order to the new semaphore in take_and_deinit.

Solving that would mean a custom Semaphore and ll_sem in tokio speak. Especially considering the new RWLock, it's really nice to be able to close the semaphore. Alternatively we could use the Inner::value as a state variable more (is_some()) and add all permits to the current semaphore (u32::MAX or so) as "closing".. Need to consider this..

EDIT: tried to implement this in #6657.

[problame]

Not sure if the init_take_deinit_scenario test cases are worth it. I would have preferred a failpoint-style approach. Obviously not with failpoint's global state, but, a cfg(test) field in the heavier_once_cell to configure the failpoint.

---

Why did we have to switch from acquire_owned() to acquire() in this PR? Would help the clarity of this change if we could avoid that.

---

Regarding the core change, I think it prevents the bug we found from occuring. 95% confident it doesn't introduce new issues.

Commit message should point out that the essence of this change is the Arc::ptr_eq.

[koivunej]

Why did we have to switch from acquire_owned() to acquire() in this PR? Would help the clarity of this change if we could avoid that.

So that the Arc<Semaphore> is not consumed and we can do the Arc::ptr_eq.

Commit message should point out that the essence of this change is the Arc::ptr_eq.

I don't think the Arc::ptr_eq is necessary here -- could just always loop when closed.