Add retry logic to EventHubAsyncCollector.SendBatchAsync in the partitioned-output branch

[Francisco-Gamino]

A customer reported some issues when writing to Event Hub for their function app. The exception points to SendBatchAsync.

After reaching out to the Event Hub team, they mentioned that this could be caused by a networking issue, so adding retry logic will help mitigate this. The fix has to go in partitioned-output branch.

Here is the full stack trace:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: <FunctionName> ---> System.InvalidOperationException : Error while handling parameter _binder after function returned: ---> System.TimeoutException : The operation did not complete within the allocated time 00:00:00 for object out-connection82.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar,Func`2 endFunction,Action`1 endAction,Task`1 promise,Boolean requiresSynchronization)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient.CreateConnectionAsync(TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync[T](TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.Amqp.Singleton`1.CreateValue[TValue](TaskCompletionSource`1 tcs,TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync[TValue](TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.CreateLinkAsync(TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync[T](TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.Amqp.Singleton`1.CreateValue[TValue](TaskCompletionSource`1 tcs,TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync[TValue](TimeSpan timeout)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.OnSendAsync(IEnumerable`1 eventDatas,String partitionKey)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.EventHubs.Amqp.AmqpEventDataSender.OnSendAsync(IEnumerable`1 eventDatas,String partitionKey)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.EventHubs.EventHubClient.SendAsync(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.EventHubs.EventHubAsyncCollector.SendBatchAsync(String partitionKey,IEnumerable`1 batch)

cc: @amamounelsayed @alrod

[alrod]

we are not going to support "partitioned-output" branch. Please create a new feature request in track2 if needed.

[alrod]

EventHubClient has the default retry policy: retry up to 9 times with an exponential back-off time of up to 30 seconds.