Uploads to Azure Blob Storage in `gh bbs2gh` only work when the archive is up to 2GB

[timrogers]

Description

A customer taking part in the private beta of Bitbucket Server migrations has reported that they are unable to migrate a repo due to the following error:

[10:38 AM] [ERROR] System.IO.IOException: The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
   at System.IO.File.ReadAllBytesAsync(String , CancellationToken )
   at OctoshiftCLI.Services.FileSystemProvider.ReadAllBytesAsync(String path)
   at OctoshiftCLI.BbsToGithub.Handlers.MigrateRepoCommandHandler.UploadArchiveToAzure(String archivePath)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& )
   at OctoshiftCLI.BbsToGithub.Handlers.MigrateRepoCommandHandler.UploadArchiveToAzure(String archivePath)
   at OctoshiftCLI.BbsToGithub.Handlers.MigrateRepoCommandHandler.Handle(MigrateRepoCommandArgs args)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at OctoshiftCLI.BbsToGithub.Handlers.MigrateRepoCommandHandler.GenerateArchive(MigrateRepoCommandArgs args)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult )
   at OctoshiftCLI.Services.BbsApi.GetExport(Int64 id)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at OctoshiftCLI.Services.BbsClient.GetAsync(String url)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at OctoshiftCLI.RetryPolicy.Retry[T](Func`1 func)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at OctoshiftCLI.Services.BbsClient.<>c__DisplayClass6_0.<<GetAsync>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at OctoshiftCLI.Services.BbsClient.SendAsync(HttpMethod httpMethod, String url, Object body)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage , HttpCompletionOption , CancellationTokenSource , Boolean , CancellationTokenSource , CancellationToken )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage , CancellationToken )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>g__Core|5_0(HttpRequestMessage , CancellationToken )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage , Boolean , CancellationToken )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage , Boolean , Boolean , CancellationToken )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage , Boolean , CancellationToken )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult()
   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter , Memory`1 )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox , Boolean )
   at System.Threading.Tasks.Task.RunContinuations(Object )
   at System.Threading.Tasks.Task`1.TrySetResult(TResult )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 , TResult )
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object )
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext , ContextCallback , Object )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread )
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 , Object , Boolean , Boolean )
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 , Byte[] , Int32 , SocketFlags , SocketError )
   at System.Net.Sockets.SocketAsyncContext.BufferMemoryReceiveOperation.InvokeCallback(Boolean )
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation )
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
   at OctoshiftCLI.BbsToGithub.Handlers.MigrateRepoCommandHandler.UploadArchiveToAzure(String archivePath)
   at OctoshiftCLI.BbsToGithub.Handlers.MigrateRepoCommandHandler.Handle(MigrateRepoCommandArgs args)
   at OctoshiftCLI.Extensions.CommandExtensions.<>c__DisplayClass1_0`3.<<ConfigureCommand>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext )
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

We should review how we are uploading files to Azure and make sure it supports files larger than 2GB, certainly up to GEI's 10GB limit.

Reproduction Steps

Attempt to migrate a repo from Bitbucket Server with bbs2gh migrate-repo where the archive is larger than 2GB. It should be possible to achieve this with a large Git repo or big attachments in PRs.

[dylan-smith]

We've already solved this problem in gei just need to make the bbs2gh code follow the same pattern.

The code here: https://github.com/github/gh-gei/blob/main/src/bbs2gh/Handlers/MigrateRepoCommandHandler.cs#L191

Should be modified to follow the pattern we have here: https://github.com/github/gh-gei/blob/main/src/gei/Handlers/MigrateRepoCommandHandler.cs#L284

The key part is using streams instead of reading the entire file into an in-memory string.

[timrogers]

I might see if I can pick this up so the customer doesn't have to wait, given that we have prior art.