fix: [precaution fix] Capture Python futures while running in the background

[kthui]

What does the PR do?

The py_future object generated by RunCoroutine() is lost upon the function returns if the coroutine is to be ran in the background. This can be problematic as some sources suggest the run_coroutine_threadsafe() only maintain a weak reference to the py_future object that it returns. Therefore, this PR will implement the workaround suggested on the official Python documentation which stores the py_future in a Python set and remove it upon the done callback is invoked, so the py_future cannot be dangling while waiting for it to execute in the background.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

N/A

Where should the reviewer start?

N/A

Test plan:

No additional tests were added. The current tests, i.e. L0_backend_python should provide sufficient coverage for any new issue introduced by the change.

• CI Pipeline ID:
  #15608296

Caveats:

We can explore if the Python library actually has weak reference while waiting for background futures.

Background

The future object returned from asyncio.run_coroutine_threadsafe(...) is not captured by the stub process while waiting for the future object to complete execution in the background. If the run_coroutine_threadsafe() only maintains a weak reference to the future object, the future object may be deleted by the time the result is ready.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

N/A

[GuanLuo]

Can you update the title / description following the PR template?