Unable to import & transform 30M documents locally with 6.0 #17448

loraderon · 2023-10-02T18:00:31Z

loraderon
Oct 2, 2023

I'm trying to import about 30M documents from a remote PostgreSQL database to my local computer with 16GB RAM and about 120GB of free disk space. I get to about 14M documents then computer freezes for a while before the failed import message appears.

Failed to enqueue batch of 3031 documents and attachments. ids: Statistics/, Statistics/, Statistics/, Statistics/, Statistics/ ...
System.InvalidOperationException: Failed to enqueue batch of 3031 documents and attachments. ids: Statistics/, Statistics/, Statistics/, Statistics/, Statistics/ ...
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Raven.Server.Documents.TransactionMerger.AbstractTransactionOperationsMerger`2.MergeTransactionsWithAsyncCommit(TOperationContext& previous, IDisposable& returnPreviousContext, List`1 previousPendingOps) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\TransactionMerger\AbstractTransactionOperationsMerger.cs:line 627
   at Raven.Server.Documents.TransactionMerger.AbstractTransactionOperationsMerger`2.MergeTransactionsOnce() in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\TransactionMerger\AbstractTransactionOperationsMerger.cs:line 423
   at Raven.Server.Documents.TransactionMerger.AbstractTransactionOperationsMerger`2.MergeOperationThreadProc() in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\TransactionMerger\AbstractTransactionOperationsMerger.cs:line 186
--- End of stack trace from previous location ---
   at Raven.Server.Documents.TransactionMerger.AbstractTransactionOperationsMerger`2.Enqueue(MergedTransactionCommand`2 cmd) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\TransactionMerger\AbstractTransactionOperationsMerger.cs:line 137
   at Raven.Server.Documents.TombstoneCleaner.ExecuteCleanup(Nullable`1 numberOfTombstonesToDeleteInBatch) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\TombstoneCleaner.cs:line 121
   --- End of inner exception stack trace ---
   at Raven.Server.SqlMigration.SqlMigrationWriter.FlushCommands() in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\SqlMigration\SqlMigrationWriter.cs:line 98
   at Raven.Client.Util.AsyncHelpers.RunSync(Func`1 task) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Client\Util\AsyncHelpers.cs:line 40
   at Raven.Server.SqlMigration.SqlMigrationWriter.Dispose() in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\SqlMigration\SqlMigrationWriter.cs:line 114
   at Raven.Server.SqlMigration.GenericDatabaseMigrator.Migrate(MigrationSettings settings, DatabaseSchema dbSchema, DocumentDatabase db, DocumentsOperationContext context, MigrationResult result, Action`1 onProgress, CancellationToken token) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\SqlMigration\GenericDatabaseMigrator.cs:line 196
   at Raven.Server.SqlMigration.GenericDatabaseMigrator.Migrate(MigrationSettings settings, DatabaseSchema dbSchema, DocumentDatabase db, DocumentsOperationContext context, MigrationResult result, Action`1 onProgress, CancellationToken token) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\SqlMigration\GenericDatabaseMigrator.cs:line 196
   at Raven.Server.SqlMigration.GenericDatabaseMigrator.Migrate(MigrationSettings settings, DatabaseSchema dbSchema, DocumentDatabase db, DocumentsOperationContext context, MigrationResult result, Action`1 onProgress, CancellationToken token) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\SqlMigration\GenericDatabaseMigrator.cs:line 196
   at Raven.Server.Web.Studio.SqlMigrationHandler.<>c__DisplayClass1_1.<<ImportSql>b__1>d.MoveNext() in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Web\Studio\SqlMigrationHandler.cs:line 80

The documents are pretty small, but I get a lot of log errors & warnings about low memory.

{
    "ChannelCampaignId": "70171",
    "Impressions": 21,
    "Media": "example.com",
    "Clicks": 2,
    "CampaignId": "329bb91c-b126-4e44-986a-9e6934221d0a",
    "Channel": 0,
    "Date": "2023-07-20",
    "Cost": 1.34,
    "Device": 4,
    "@metadata": {
        "@collection": "Statistics"
    }
}

2023-10-02T17:24:56.3218112Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.735 GBytes / 63.771 GBytes, Memory: 15.358 GBytes / 15.771 GBytes, Available memory for processing: 422.6 MBytes, Dirty memory: 0 Bytes, Managed memory: 437.68 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:24:59.3263752Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.733 GBytes / 63.771 GBytes, Memory: 15.35 GBytes / 15.771 GBytes, Available memory for processing: 430.81 MBytes, Dirty memory: 0 Bytes, Managed memory: 436.52 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:02.3265803Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.731 GBytes / 63.771 GBytes, Memory: 15.347 GBytes / 15.771 GBytes, Available memory for processing: 433.82 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.83 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:05.3321132Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.739 GBytes / 63.771 GBytes, Memory: 15.349 GBytes / 15.771 GBytes, Available memory for processing: 431.64 MBytes, Dirty memory: 0 Bytes, Managed memory: 468.09 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:08.3373486Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.748 GBytes / 63.771 GBytes, Memory: 15.357 GBytes / 15.771 GBytes, Available memory for processing: 423.52 MBytes, Dirty memory: 0 Bytes, Managed memory: 451.33 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:11.3450788Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.754 GBytes / 63.771 GBytes, Memory: 15.363 GBytes / 15.771 GBytes, Available memory for processing: 417.72 MBytes, Dirty memory: 0 Bytes, Managed memory: 474.6 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:14.3482527Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.753 GBytes / 63.771 GBytes, Memory: 15.365 GBytes / 15.771 GBytes, Available memory for processing: 415.74 MBytes, Dirty memory: 0 Bytes, Managed memory: 463.86 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:16.6562332Z, 39, Operations, System, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.758 GBytes / 63.771 GBytes, Memory: 15.368 GBytes / 15.771 GBytes, Available memory for processing: 412.7 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.86 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:25:16.6565847Z, 30, Operations, spiderads, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.758 GBytes / 63.771 GBytes, Memory: 15.368 GBytes / 15.771 GBytes, Available memory for processing: 412.7 MBytes, Dirty memory: 0 Bytes, Managed memory: 465.09 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:25:17.3490892Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.76 GBytes / 63.771 GBytes, Memory: 15.369 GBytes / 15.771 GBytes, Available memory for processing: 410.87 MBytes, Dirty memory: 0 Bytes, Managed memory: 475.73 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:20.3569376Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.764 GBytes / 63.771 GBytes, Memory: 15.375 GBytes / 15.771 GBytes, Available memory for processing: 405.52 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.84 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:22.8410005Z, 12, Operations, ServerStore, Raven.Server.ServerWide.ServerStore, Error during idle operations for the server, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.739 GBytes / 63.771 GBytes, Memory: 15.363 GBytes / 15.771 GBytes, Available memory for processing: 416.93 MBytes, Dirty memory: 0 Bytes, Managed memory: 494.3 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.LowLevelTransaction..ctor(StorageEnvironment env, Int64 id, TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, IFreeSpaceHandling freeSpaceHandling, ByteStringContext context) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\LowLevelTransaction.cs:line 349
   at Voron.StorageEnvironment.NewLowLevelTransaction(TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, ByteStringContext context, Nullable`1 timeout) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\StorageEnvironment.cs:line 767
   at Voron.StorageEnvironment.Stats() in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\StorageEnvironment.cs:line 1450
   at Raven.Server.Documents.DatabasesLandlord.LastWork(DocumentDatabase resource) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\DatabasesLandlord.cs:line 1204
   at Raven.Server.ServerWide.ServerStore.CanUnloadDatabase(StringSegment databaseName, DateTime lastRecentlyUsed, IdleDatabaseStatistics statistics, DocumentDatabase& database) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\ServerWide\ServerStore.cs:line 2821
   at Raven.Server.ServerWide.ServerStore.IdleOperations(Object state) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\ServerWide\ServerStore.cs:line 2712
2023-10-02T17:25:23.3664903Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.738 GBytes / 63.771 GBytes, Memory: 15.361 GBytes / 15.771 GBytes, Available memory for processing: 419.38 MBytes, Dirty memory: 0 Bytes, Managed memory: 462.66 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 1.057 GBytes, managed thread id: 10, [#4] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:26.4476965Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.742 GBytes / 63.771 GBytes, Memory: 15.256 GBytes / 15.771 GBytes, Available memory for processing: 526.6 MBytes, Dirty memory: 0 Bytes, Managed memory: 464 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:29.5260795Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.743 GBytes / 63.771 GBytes, Memory: 15.256 GBytes / 15.771 GBytes, Available memory for processing: 527.02 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.11 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:32.5922653Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.741 GBytes / 63.771 GBytes, Memory: 15.262 GBytes / 15.771 GBytes, Available memory for processing: 520.79 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.14 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:35.6547143Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.74 GBytes / 63.771 GBytes, Memory: 15.26 GBytes / 15.771 GBytes, Available memory for processing: 522.35 MBytes, Dirty memory: 0 Bytes, Managed memory: 463.99 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:38.7325920Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.735 GBytes / 63.771 GBytes, Memory: 15.26 GBytes / 15.771 GBytes, Available memory for processing: 523.03 MBytes, Dirty memory: 0 Bytes, Managed memory: 463.87 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:41.7972514Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.725 GBytes / 63.771 GBytes, Memory: 15.255 GBytes / 15.771 GBytes, Available memory for processing: 527.52 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.3 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:44.9209681Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.724 GBytes / 63.771 GBytes, Memory: 15.255 GBytes / 15.771 GBytes, Available memory for processing: 528.38 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.2 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:46.6588700Z, 39, Operations, spiderads, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.725 GBytes / 63.771 GBytes, Memory: 15.255 GBytes / 15.771 GBytes, Available memory for processing: 527.76 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.13 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:25:46.6597090Z, 14, Operations, System, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.725 GBytes / 63.771 GBytes, Memory: 15.255 GBytes / 15.771 GBytes, Available memory for processing: 527.76 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.18 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:25:47.9240008Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.725 GBytes / 63.771 GBytes, Memory: 15.255 GBytes / 15.771 GBytes, Available memory for processing: 528.41 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.42 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:50.9846235Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 33 low memory handlers with severity: ExtremelyLow. Commit charge: 63.723 GBytes / 63.771 GBytes, Memory: 15.254 GBytes / 15.771 GBytes, Available memory for processing: 528.56 MBytes, Dirty memory: 0 Bytes, Managed memory: 464.08 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:54.1266226Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.763 GBytes / 63.771 GBytes, Memory: 14.125 GBytes / 15.771 GBytes, Available memory for processing: 1.646 GBytes, Dirty memory: 0 Bytes, Managed memory: 464.52 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:25:57.2948430Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 35 low memory handlers with severity: ExtremelyLow. Commit charge: 63.761 GBytes / 63.771 GBytes, Memory: 14.123 GBytes / 15.771 GBytes, Available memory for processing: 1.647 GBytes, Dirty memory: 0 Bytes, Managed memory: 465 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:00.6861558Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 37 low memory handlers with severity: ExtremelyLow. Commit charge: 63.763 GBytes / 63.771 GBytes, Memory: 14.128 GBytes / 15.771 GBytes, Available memory for processing: 1.642 GBytes, Dirty memory: 0 Bytes, Managed memory: 465.47 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:03.7654499Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 39 low memory handlers with severity: ExtremelyLow. Commit charge: 63.761 GBytes / 63.771 GBytes, Memory: 14.128 GBytes / 15.771 GBytes, Available memory for processing: 1.643 GBytes, Dirty memory: 0 Bytes, Managed memory: 465.86 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:06.8472683Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 39 low memory handlers with severity: ExtremelyLow. Commit charge: 63.761 GBytes / 63.771 GBytes, Memory: 14.126 GBytes / 15.771 GBytes, Available memory for processing: 1.644 GBytes, Dirty memory: 0 Bytes, Managed memory: 466.27 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:09.9043418Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 41 low memory handlers with severity: ExtremelyLow. Commit charge: 63.762 GBytes / 63.771 GBytes, Memory: 14.132 GBytes / 15.771 GBytes, Available memory for processing: 1.639 GBytes, Dirty memory: 0 Bytes, Managed memory: 466.71 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:13.0129728Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 43 low memory handlers with severity: ExtremelyLow. Commit charge: 63.759 GBytes / 63.771 GBytes, Memory: 14.132 GBytes / 15.771 GBytes, Available memory for processing: 1.639 GBytes, Dirty memory: 0 Bytes, Managed memory: 467.14 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:16.0962274Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 43 low memory handlers with severity: ExtremelyLow. Commit charge: 63.759 GBytes / 63.771 GBytes, Memory: 14.131 GBytes / 15.771 GBytes, Available memory for processing: 1.639 GBytes, Dirty memory: 0 Bytes, Managed memory: 467.57 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:16.6677032Z, 30, Operations, System, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.759 GBytes / 63.771 GBytes, Memory: 14.131 GBytes / 15.771 GBytes, Available memory for processing: 1.639 GBytes, Dirty memory: 0 Bytes, Managed memory: 467.75 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:26:16.6678977Z, 3, Operations, spiderads, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.759 GBytes / 63.771 GBytes, Memory: 14.131 GBytes / 15.771 GBytes, Available memory for processing: 1.639 GBytes, Dirty memory: 0 Bytes, Managed memory: 467.86 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:26:19.1638536Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 45 low memory handlers with severity: ExtremelyLow. Commit charge: 63.77 GBytes / 63.771 GBytes, Memory: 14.22 GBytes / 15.771 GBytes, Available memory for processing: 1.551 GBytes, Dirty memory: 0 Bytes, Managed memory: 468.26 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:22.2175412Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 45 low memory handlers with severity: ExtremelyLow. Commit charge: 63.737 GBytes / 63.771 GBytes, Memory: 14.205 GBytes / 15.771 GBytes, Available memory for processing: 1.565 GBytes, Dirty memory: 0 Bytes, Managed memory: 468.68 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:22.8511661Z, 3, Operations, ServerStore, Raven.Server.ServerWide.ServerStore, Error during idle operations for the server, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.737 GBytes / 63.771 GBytes, Memory: 14.205 GBytes / 15.771 GBytes, Available memory for processing: 1.566 GBytes, Dirty memory: 0 Bytes, Managed memory: 468.79 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.LowLevelTransaction..ctor(StorageEnvironment env, Int64 id, TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, IFreeSpaceHandling freeSpaceHandling, ByteStringContext context) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\LowLevelTransaction.cs:line 349
   at Voron.StorageEnvironment.NewLowLevelTransaction(TransactionPersistentContext transactionPersistentContext, TransactionFlags flags, ByteStringContext context, Nullable`1 timeout) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\StorageEnvironment.cs:line 767
   at Voron.StorageEnvironment.Stats() in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\StorageEnvironment.cs:line 1450
   at Raven.Server.Documents.DatabasesLandlord.LastWork(DocumentDatabase resource) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\Documents\DatabasesLandlord.cs:line 1204
   at Raven.Server.ServerWide.ServerStore.CanUnloadDatabase(StringSegment databaseName, DateTime lastRecentlyUsed, IdleDatabaseStatistics statistics, DocumentDatabase& database) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\ServerWide\ServerStore.cs:line 2821
   at Raven.Server.ServerWide.ServerStore.IdleOperations(Object state) in C:\Builds\RavenDB-Stable-6.0\60002\src\Raven.Server\ServerWide\ServerStore.cs:line 2712
2023-10-02T17:26:25.3054905Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 47 low memory handlers with severity: ExtremelyLow. Commit charge: 63.737 GBytes / 63.771 GBytes, Memory: 14.215 GBytes / 15.771 GBytes, Available memory for processing: 1.555 GBytes, Dirty memory: 0 Bytes, Managed memory: 469.15 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:28.3922494Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 49 low memory handlers with severity: ExtremelyLow. Commit charge: 63.738 GBytes / 63.771 GBytes, Memory: 14.197 GBytes / 15.771 GBytes, Available memory for processing: 1.574 GBytes, Dirty memory: 0 Bytes, Managed memory: 469.65 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:31.4684673Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 49 low memory handlers with severity: ExtremelyLow. Commit charge: 63.738 GBytes / 63.771 GBytes, Memory: 14.196 GBytes / 15.771 GBytes, Available memory for processing: 1.574 GBytes, Dirty memory: 0 Bytes, Managed memory: 470.07 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:35.3456916Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 56 low memory handlers with severity: ExtremelyLow. Commit charge: 63.595 GBytes / 63.771 GBytes, Memory: 14.46 GBytes / 15.771 GBytes, Available memory for processing: 1.311 GBytes, Dirty memory: 0 Bytes, Managed memory: 471.43 MBytes, Unmanaged allocations: 53.312 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.36 MBytes, managed thread id: 14
2023-10-02T17:26:38.4807700Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 56 low memory handlers with severity: ExtremelyLow. Commit charge: 63.736 GBytes / 63.771 GBytes, Memory: 14.424 GBytes / 15.771 GBytes, Available memory for processing: 1.347 GBytes, Dirty memory: 0 Bytes, Managed memory: 471.67 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:41.9963918Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 63 low memory handlers with severity: ExtremelyLow. Commit charge: 63.757 GBytes / 63.771 GBytes, Memory: 14.523 GBytes / 15.771 GBytes, Available memory for processing: 1.248 GBytes, Dirty memory: 0 Bytes, Managed memory: 472.7 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:45.8482442Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 68 low memory handlers with severity: ExtremelyLow. Commit charge: 63.687 GBytes / 63.771 GBytes, Memory: 14.525 GBytes / 15.771 GBytes, Available memory for processing: 1.246 GBytes, Dirty memory: 0 Bytes, Managed memory: 473.79 MBytes, Unmanaged allocations: 53.312 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.34 MBytes, managed thread id: 14
2023-10-02T17:26:48.9487372Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 68 low memory handlers with severity: ExtremelyLow. Commit charge: 63.722 GBytes / 63.771 GBytes, Memory: 14.56 GBytes / 15.771 GBytes, Available memory for processing: 1.21 GBytes, Dirty memory: 0 Bytes, Managed memory: 474.09 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:51.6600193Z, 12, Operations, spiderads, Voron.Impl.Journal.WriteAheadJournal, Could not allocate enough space to apply pages to data file, EXCEPTION: Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 63.757 GBytes / 63.771 GBytes, Memory: 14.541 GBytes / 15.771 GBytes, Available memory for processing: 1.23 GBytes, Dirty memory: 0 Bytes, Managed memory: 475.27 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
   at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 193
   at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 479
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 623
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 1410
   at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in C:\Builds\RavenDB-Stable-6.0\60002\src\Sparrow.Server\ByteString.cs:line 786
   at Voron.Impl.Journal.WriteAheadJournal.JournalApplicator.ApplyLogsToDataFile(CancellationToken token, TimeSpan timeToWait) in C:\Builds\RavenDB-Stable-6.0\60002\src\Voron\Impl\Journal\WriteAheadJournal.cs:line 859
2023-10-02T17:26:52.0698093Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 75 low memory handlers with severity: ExtremelyLow. Commit charge: 63.739 GBytes / 63.771 GBytes, Memory: 14.535 GBytes / 15.771 GBytes, Available memory for processing: 1.236 GBytes, Dirty memory: 0 Bytes, Managed memory: 475.34 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:55.2537461Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 77 low memory handlers with severity: ExtremelyLow. Commit charge: 63.726 GBytes / 63.771 GBytes, Memory: 14.527 GBytes / 15.771 GBytes, Available memory for processing: 1.244 GBytes, Dirty memory: 0 Bytes, Managed memory: 475.67 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:26:58.3836205Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 77 low memory handlers with severity: ExtremelyLow. Commit charge: 63.703 GBytes / 63.771 GBytes, Memory: 14.51 GBytes / 15.771 GBytes, Available memory for processing: 1.261 GBytes, Dirty memory: 0 Bytes, Managed memory: 476.18 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:27:01.4999632Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 79 low memory handlers with severity: ExtremelyLow. Commit charge: 63.703 GBytes / 63.771 GBytes, Memory: 14.508 GBytes / 15.771 GBytes, Available memory for processing: 1.263 GBytes, Dirty memory: 0 Bytes, Managed memory: 476.64 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:27:04.6969793Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 83 low memory handlers with severity: ExtremelyLow. Commit charge: 63.758 GBytes / 63.771 GBytes, Memory: 14.616 GBytes / 15.771 GBytes, Available memory for processing: 1.155 GBytes, Dirty memory: 0 Bytes, Managed memory: 477.61 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14
2023-10-02T17:27:07.8968675Z, 15, Operations, Server, Sparrow.LowMemory.LowMemoryNotification, Running 93 low memory handlers with severity: ExtremelyLow. Commit charge: 63.764 GBytes / 63.771 GBytes, Memory: 14.61 GBytes / 15.771 GBytes, Available memory for processing: 1.161 GBytes, Dirty memory: 0 Bytes, Managed memory: 478.59 MBytes, Unmanaged allocations: 53.311 GBytes, Lucene managed: 0 Bytes, Lucene unmanaged: 0 Bytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 7.199 GBytes, managed thread id: 40, [#2] name: .NET ThreadPool Worker, allocations: 1.687 GBytes, managed thread id: 39, [#3] name: .NET ThreadPool Worker, allocations: 347.25 MBytes, managed thread id: 14

Answered by ArieSLV

Oct 3, 2023

Fixes for both points have been merged and will be available in the next stable build.

Again, thank you for the provided information, and I am sorry about any possible inconvenience.

View full answer

georgiosd · 2023-10-02T18:56:25Z

georgiosd
Oct 2, 2023

Did you try increasing your swap file?

1 reply

loraderon Oct 2, 2023
Author

No, but it seems to work much better after I removed the following transform:

this.Date = this.Date.substring(0, 10)
delete this["@metadata"]["@sql-keys"]

The date column in pg is defined as date date NOT NULL so it does not contain any null values.

DejanMilicic · 2023-10-02T19:07:05Z

DejanMilicic
Oct 2, 2023
Collaborator

Can you create a sample which would help us reproduce this and fix it?

4 replies

loraderon Oct 2, 2023
Author

I'll keep investigating, but it seems at the moment be related to the "Transform documents" script I posted above.
All 30 647 366 documents imported sucessfully without it.

It takes a while to import, delete and try again. So my feedback loop is pretty slow. I'll let you know how it goes and if I can reproduce it faster without this amount of data.

loraderon Oct 2, 2023
Author

I was able to create a index with the date substring and it managed to index all of the documents.

map("Statistics", (stat) => {
    return {
        DateOnly: stat.Date.substring(0, 10)
    };
})

Next I'll try to import the documents with a transform script that doesn't try to delete the (empty) "@sql-keys". But it's getting late and I'll have to continue this tomorrow. Looks like it's not an issue with handling 30M documents on a laptop - and that is great!

ayende Oct 2, 2023
Maintainer

I wonder if it is possible that we aren't disposing the transformed entity along the way

loraderon Oct 3, 2023
Author

I have tried the import with only this.Date = this.Date.substring(0, 10) and with only delete this["@metadata"]["@sql-keys"] and it got to 12M and 13M documents before failure.

Tried again without any transform script and then it completes without issues or warnings.

Transform scripts are not available for CSV imports so I cannot test if the errors are related to the remote connection (and the extra processing overhead).

Is this information enough for you to investigate it? I do not think there is anything special with the records since they import fine without the transform script.

ArieSLV · 2023-10-03T08:09:20Z

ArieSLV
Oct 3, 2023
Collaborator

@loraderon hi,
Thank you for the information provided.

We have opened two issues based on this information:

https://issues.hibernatingrhinos.com/issue/RavenDB-21517/Import-from-SQL-with-a-transform-script-causes-high-usage-of-unmanaged-memory
https://issues.hibernatingrhinos.com/issue/RavenDB-21516/NRE-in-transaction-merger-when-failing-to-allocate-a-context (fix has already been merged and will be available in the next stable build)

3 replies

loraderon Oct 3, 2023
Author

Thank you, I'll verify it again when both of these issues are fixed in a build that I can download.

ArieSLV Oct 3, 2023
Collaborator

The reason for the high level of unmanaged memory allocations (first point) has been found as well, the fix is in the pull request stage: #17457

ArieSLV Oct 3, 2023
Collaborator

Fixes for both points have been merged and will be available in the next stable build.

Again, thank you for the provided information, and I am sorry about any possible inconvenience.

Answer selected by ArieSLV

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to import & transform 30M documents locally with 6.0 #17448

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Unable to import & transform 30M documents locally with 6.0 #17448

loraderon Oct 2, 2023

Replies: 3 comments · 8 replies

georgiosd Oct 2, 2023

loraderon Oct 2, 2023 Author

DejanMilicic Oct 2, 2023 Collaborator

loraderon Oct 2, 2023 Author

loraderon Oct 2, 2023 Author

ayende Oct 2, 2023 Maintainer

loraderon Oct 3, 2023 Author

ArieSLV Oct 3, 2023 Collaborator

loraderon Oct 3, 2023 Author

ArieSLV Oct 3, 2023 Collaborator

ArieSLV Oct 3, 2023 Collaborator

loraderon
Oct 2, 2023

Replies: 3 comments 8 replies

georgiosd
Oct 2, 2023

loraderon Oct 2, 2023
Author

DejanMilicic
Oct 2, 2023
Collaborator

loraderon Oct 2, 2023
Author

loraderon Oct 2, 2023
Author

ayende Oct 2, 2023
Maintainer

loraderon Oct 3, 2023
Author

ArieSLV
Oct 3, 2023
Collaborator

loraderon Oct 3, 2023
Author

ArieSLV Oct 3, 2023
Collaborator

ArieSLV Oct 3, 2023
Collaborator