[Core]Fix actor creation race condition of #59642

[YoyinZyc]

Description

Run and clear the actor creation callback before actor recreation to avoid race condition.

A race condition occurs when an actor is successfully created on a worker, but the worker dies before GCS completes the asynchronous write of the actor's ALIVE state to storage. While waiting for the storage write, GCS processes the worker death and clears the actor's address in memory. When the storage write finally completes, its callback reads the cleared (Nil) address and sends it to the client, causing a crash.

With the suggestion from @MengjinYan, in RestartActor, before clearing the actor's address in memory, we check if the actor was already successfully created (ALIVE) and has pending creation callbacks. If so, we invoke the callbacks early with the valid address still in memory and the stored borrowed_refs. This ensures the client receives a valid address and avoids the crash.

Related issues

Fixes #59642

Tested

• added a new unit test
• reproduce the issue and verify the fix

[gemini-code-assist]

Code Review

This pull request introduces functionality to store and retrieve borrowed references for actors within the GCS. It also updates the RestartActor logic to trigger pending creation callbacks early using these saved references. A critical review comment points out that these callbacks should only be invoked with a success status if the actor had previously reached the ALIVE state; otherwise, it might incorrectly signal to the creator that the actor is ready while it is still restarting from a pending state.

[andrewsykim]

Thanks for the PR @YoyinZyc, can you try to run the reproduction script shared in the original issue (#59642) with your changes?

[YoyinZyc]

Thanks for the PR @YoyinZyc, can you try to run the reproduction script shared in the original issue (#59642) with your changes?

Yes I reproduced the issue like what suggested in the issue and verified the change works.

• Injected the failure by delaying GCS storage write
• kill the worker after actor recreation

Verified

• No nil address crash
• "Invoking creation callbacks early in RestartActor" in gcs log.

[MengjinYan]

Thanks for the PR, @YoyinZyc!

I have 2 general comments:

• In the PR description, can you add a bit more context about the issue itself and the changes you made so that people can get more context of the change when they read the PR?
• Wondering is it possible to add a test to verify the situation of the issue?

[YoyinZyc]

@MengjinYan thanks for the suggestion.

Updated the pr description and added a new unit test.

[YoyinZyc]

Friendly ping on this issue:) @MengjinYan @andrewsykim

[MengjinYan]

Thanks for the fix!! All are nit comments.

Also, it seems that the lint is failing. Probably also need to fix that

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 1e3df78. Configure here.}

[MengjinYan]

Thanks!