[Bug] Client hangs when waiting for calls on main thread

[cretz]

Describe the bug

To replicate, make a project with this:

using Temporalio.Client;
using Temporalio.Client.Schedules;
using Temporalio.Runtime;
using Temporalio.Workflows;

// Connect client
var runtime = new TemporalRuntime(new()
{
    Telemetry = new()
    {
        Logging = new()
        {
            // Filter = new(TelemetryFilterOptions.Level.Trace, TelemetryFilterOptions.Level.Trace),
        }
    }
});
var client = await TemporalClient.ConnectAsync(new("localhost:7233") { Runtime = runtime });

// Two schedule tasks
var tasks = new List<Task<ScheduleHandle>>
{
    CreateScheduleAsync(),
    CreateScheduleAsync(),
};

// Running Task.WaitAll hangs, but Task.WaitAll off the main thread or Task.WhenAll works

// DOES NOT WORK:
Task.WaitAll(tasks.ToArray());

// WORKS:
// await Task.Run(() => Task.WaitAll(tasks.ToArray()));

// WORKS:
// await Task.WhenAll(tasks.ToArray());

foreach (var task in tasks)
{
    var desc = task.Result.DescribeAsync();
    Console.WriteLine("Described schedule: {0}", desc.Id);
}

async Task<ScheduleHandle> CreateScheduleAsync()
{
    var action = ScheduleActionStartWorkflow.Create<MyWorkflow>(
        wf => wf.RunAsync(),
        new()
        {
            Id = $"my-workflow-{Guid.NewGuid()}",
            TaskQueue = "my-task-queue-{Guid.NewGuid()}",
        });
    var spec = new ScheduleSpec()
    {
        Intervals = new List<ScheduleIntervalSpec> { new(TimeSpan.FromHours(10)) },
    };
    var id = $"my-schedule-{Guid.NewGuid()}";
    Console.WriteLine("Starting schedule: {0}", id);
    var handle = await client.CreateScheduleAsync(id, new(action, spec));
    Console.WriteLine("Schedule started: {0}", id);
    return handle;
}

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync() => "done";
}

An easy way to do this is to update tests/Temporalio.SmokeTest/Program.cs with this and add this in the `Temporalio.SmokeTest.csproj:

  <ItemGroup>
    <ProjectReference Include="..\..\src\Temporalio\Temporalio.csproj" />
  </ItemGroup>

Then in that directory, run dotnet run. Note, sdk-core can be updated and dotnet build can be run to recompile if doing guess-and-check debugging.

What is happening is that Tonic seems to hang when this is waited on the main thread. I have hooked up Tokio console and I see nothing obvious. I have upgraded, turned on tracing, etc and see nothing obvious. A very similar thing happened in #44 but the fix was a core/dependency upgrade yet we could not find which bug fixed it upstream in Tonic/h2/hyper/tokio.

[cretz]

Thanks to @robcao, the TaskCompletionSource can deadlock on same thread, need to supply TaskCreationOptions.RunContinuationsAsynchronously to the constructor so that it will schedule it elsewhere.