PubSub: Unobserved InvalidOperationException in AsyncSingleRecvQueue

[lkts]

Hi,

My application that uses PubSub has logging handler for TaskScheduler.UnobservedTaskException.
Recently i have found out that such exceptions are constantly produced with stacktrace leading to PubSub (can be up to hundred in 10 minutes).
Exception:

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. 
As a result, the unobserved exception was rethrown by the finalizer thread. (An attempt was made to transition a task to a final state when it had already completed.) 
---> System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
   at Google.Cloud.PubSub.V1.SubscriberClientImpl.AsyncSingleRecvQueue`1.Enqueue(T item)
   at Google.Cloud.PubSub.V1.SubscriberClientImpl.SingleChannel.<>c__DisplayClass39_0.<Add>b__0(Task _)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
   at Google.Cloud.PubSub.V1.SubscriberClientImpl.AsyncSingleRecvQueue`1.Enqueue(T item)
   at Google.Cloud.PubSub.V1.SubscriberClientImpl.SingleChannel.<>c__DisplayClass39_0.<Add>b__0(Task _)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)<---

Looks like race in code below since it is not in lock.

google-cloud-dotnet/apis/Google.Cloud.PubSub.V1/Google.Cloud.PubSub.V1/SubscriberClient.cs

Line 663 in 2dc1fb7

tcs.SetResult(0);

I believe there are no functional bugs because of that but ideally library should not produce such things :)

Thanks.

[jskeet]

Thanks for reporting this! (I'm currently looking at unobserved task exceptions in Spanner, but I don't expect it's caused by the same issue.)