System.OutOfMemoryException when reading large JSON files

[RichMercer]

When using the JSON source, a System.OutOfMemoryException is thrown when reading the file. It seems like the code is trying to load the entire JSON file into memory before running the import, so for the 2GB file I have, it's throwing the exception.

Platform: MacBook Pro M1

Stack Trace:

Cosmos.DataTransfer.Core.RunCommand.CommandHandler[0]
Data transfer failed
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at System.IO.File.InternalReadAllTextAsync(String path, Encoding encoding, CancellationToken cancellationToken)
at Cosmos.DataTransfer.JsonExtension.JsonDataSourceExtension.ReadFileAsync(String filePath, ILogger logger, CancellationToken cancellationToken) in /Users/rich/GitHub/data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSourceExtension.cs:line 90
at Cosmos.DataTransfer.JsonExtension.JsonDataSourceExtension.ReadAsync(IConfiguration config, ILogger logger, CancellationToken cancellationToken)+MoveNext() in /Users/rich/GitHub/data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSourceExtension.cs:line 26
at Cosmos.DataTransfer.JsonExtension.JsonDataSourceExtension.ReadAsync(IConfiguration config, ILogger logger, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.Linq.AsyncEnumerable.SelectEnumerableAsyncIterator2.MoveNextCore() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:line 221 at System.Linq.AsyncIteratorBase1.MoveNextAsync() in //Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70
at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75 at System.Linq.AsyncEnumerable.WhereEnumerableAsyncIterator1.MoveNextCore() in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Where.cs:line 233
at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 70 at System.Linq.AsyncIteratorBase1.MoveNextAsync() in //Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 75
at System.Linq.AsyncEnumerable.g__Core|259_0[TResult](IAsyncEnumerable1 source, CancellationToken cancellationToken)+MoveNext() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/OfType.cs:line 37 at System.Linq.AsyncEnumerable.<OfType>g__Core|259_0[TResult](IAsyncEnumerable1 source, CancellationToken cancellationToken)+MoveNext() in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/OfType.cs:line 37
at System.Linq.AsyncEnumerable.g__Core|259_0[TResult](IAsyncEnumerable1 source, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at System.Linq.AsyncEnumerableEx.<Buffer>g__Core|4_0[TSource](IAsyncEnumerable1 source, Int32 count, CancellationToken cancellationToken)+MoveNext() in /_/Ix.NET/Source/System.Interactive.Async/System/Linq/Operators/Buffer.cs:line 35
at System.Linq.AsyncEnumerableEx.g__Core|4_0[TSource](IAsyncEnumerable1 source, Int32 count, CancellationToken cancellationToken)+MoveNext() in /_/Ix.NET/Source/System.Interactive.Async/System/Linq/Operators/Buffer.cs:line 35 at System.Linq.AsyncEnumerableEx.<Buffer>g__Core|4_0[TSource](IAsyncEnumerable1 source, Int32 count, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Cosmos.DataTransfer.CosmosExtension.CosmosDataSinkExtension.WriteAsync(IAsyncEnumerable1 dataItems, IConfiguration config, IDataSourceExtension dataSource, ILogger logger, CancellationToken cancellationToken) in /Users/rich/GitHub/data-migration-desktop-tool/Extensions/Cosmos/Cosmos.DataTransfer.CosmosExtension/CosmosDataSinkExtension.cs:line 111 at Cosmos.DataTransfer.CosmosExtension.CosmosDataSinkExtension.WriteAsync(IAsyncEnumerable1 dataItems, IConfiguration config, IDataSourceExtension dataSource, ILogger logger, CancellationToken cancellationToken) in /Users/rich/GitHub/data-migration-desktop-tool/Extensions/Cosmos/Cosmos.DataTransfer.CosmosExtension/CosmosDataSinkExtension.cs:line 111
at Cosmos.DataTransfer.Core.RunCommand.CommandHandler.ExecuteDataTransferOperation(IDataSourceExtension source, IConfigurationSection sourceConfig, IDataSinkExtension sink, IConfigurationSection sinkConfig, CancellationToken cancellationToken) in /Users/rich/GitHub/data-migration-desktop-tool/Core/Cosmos.DataTransfer.Core/RunCommand.cs:line 134

[bowencode]

Included in Release v2.0.3