[BUG] timeout error for other coroutines if task is synchronous function that takes too long to run

[RogerHYang]

Below is the errors you'll see in the guidlines_eval.ipynb notebook.

The root cause of this is that the task is defined to be synchronous and its network call is blocking the execution long enough to cause some of the await post to the phoenix server to be timed out.

To reproduce, use time.sleep. See below.

Note that the Phoenix server received all 7 traces in the above screenshot, but not all 7 experiment runs.