BulkInsert retry logic

[jesuslpm]

Does bulk insert operation have builtin retry logic?

If not, how to implement it?

It seems it's hard because if one batch fails you can't resend it because you don't have information about the batch, you don't know which documents belong to the batch, so you cannot dispose the bulk insert operation and store the documents into the bulk insert operation again.

Imagine you need to insert one million documents and one batch fails in de middle due to a network hiccup. What can you do? Start from the beginning? Try to resend the batch?

[ppekrol]

Hi @jesuslpm

Bulk Insert does not support retrying, you would have to start a new one yourself. There are certain limitations when it comes to this operation that you should take into account: https://ravendb.net/docs/article-page/5.3/csharp/client-api/bulk-insert/how-to-work-with-bulk-insert-operation#limitations

Also 'SkipOverwriteIfUnchanged' option might be interesting for you: https://ravendb.net/docs/article-page/5.3/csharp/client-api/bulk-insert/how-to-work-with-bulk-insert-operation#skipoverwriteifunchanged

[jesuslpm]

Hi @ppekrol.

I can start a new bulk insert operation myself, sure. But I don't want to start it from the beginning, I would prefer to start it from the last successfully inserted document. It would be nice I you raised an event or call a callback for each successfully completed batch with the batch number and the number of documents in the batch, just like the information shown on the Studio for in progress bulk insert operations.

[ppekrol]

Have you considered splitting your operation into few smaller bulk inserts? If you will successfully dispose BulkInsert then the data is written to the disk so you can use last document ID that you inserted there as your starting point if something goes wrong.

[jesuslpm]

@ppekrol I actually tested it. My documents are about 2kb in size. On the other hand I believe max batch size is 1Mb source code. So I tested with 500 documents per bulk insert operation, they fit in one batch. I got 4000 docs/second. On the other hand, using Session instead and 500 documents per batch, got 5000 docs/second. I also tested with 1000 documents per bulk insert operation, got 8000 docs/second. 2000 docs per operation: 10000 docs/second. Single bulk insert operation 11000 docs/second.

The use case is a .net core logging provider. Loggin.Raven

I wanted to keep the minimum number of documents in memory for retrying, I don't want to provide a memory eater, garbage collector unfriendly logging provider. So I think the optimal solution is BulkInsertOperation.BatchCompleted event. Because in that case I would need to maintain at much as 512 documents in memory and get 11000 docs/second

[ppekrol]

I've created an issue for this: https://issues.hibernatingrhinos.com/issue/RavenDB-18643 BulkInsertPrgress has LastProcessedId available. Should be enough for your case.

[trevhunter]

@jesuslpm - FWIW, We ran into this issue in the past - turns out there's a lot of reasons why retrying bulk inserts is problematic. @ppekrol you may want to look at https://issues.hibernatingrhinos.com/issue/RavenDB-17740 for history on the discussion.

For us though, it turns out that normal document inserts using the session.StoreAsync() and inserting around 1000-5000 documents per call to session.SaveChanges() didn't result in too much of a performance overhead compared to bulk insert and gave us everything we needed in terms of ability to retry.

Quote from our original ticket on this:

The real moral of the story for us... We'll treat the bulk insert API as a best effort load for ideal and short jobs. But where long running and reliability are needs, we'll use the session as normal since there isn't much perf tradeoffs for all the other goodness session brings.

[jesuslpm]

@ppekrol and @trevhunter. Thank you very much

[georgiosd]

Hey @jesuslpm, I wonder if you've ran any tests on this re storage + query performance vs Loki or something else?