-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
batchWrite unreliable #20
Comments
Hi, thanks for the thorough research! you're indeed right about the current design flaw. or at least, lack of feature. Bad exception management Requests not throttled let me know, thanks. |
Sometimes I use provisioned capacity (mainly without auto-scaling) because, in some circumstances, the cost must be exactly known in advance (customers require it). Anyway, unless explicitly asking AWS to increase the on-demand capacity (default is 40K read\write) this limit will be hit very quickly with the current implementation of the batchWrite. I have just tried with a set of 100K items (region: eu-west-1) and about 30% of the records are not being written. During the last week I wrote about 100M records to an on-demand table, and this is quite common in enterprise migrations (at least on my experience); without the changes made in the gist that I have linked before it would have been a nightmare to handle. So I think the best approach to quickly solve this issue and make batch writes reliable is the b option:
coupled with some sort of throttling which users should be able to customize accordingly to the provisioned capacity to minimize unprocessed items (the one in the gist is the quickest way which I came up with, but I am a javascript beginner). A more elegant\reliable approach would an exponential backoff implementation but I think that it could be added later. |
Hi, thanks for the explanation. First, On usecases where you "need to" throttle, (based on customers requirements to know budget i guess?) Second, For cases where even with on-demand but meeting limit, i completely aware of those 2 cases (account limit / hot partition)
when you have max capcity of X but you want to insert Y in time T, where (Y/T) > X, So i think
kind of implementation is better for developers.
in the meantime, if you need this urgently handled on dynamorm, i suggest doing something like
this ensures doing maximum concurrent batch of 10 for given moment (where each batch is 25) - essentially, this will be very similar with the implementation I'll create for batchWrite. thanks. |
I have temporarily solved the bulk load of millions of record with the code of the previous gist which worked like a charm. Thanks for your support and explanation, is really appreciated. |
oh yeah that's better - bluebird is indeed too heavy; |
@dawolf-tty @breath103 I've just started digging into DymamoDB and I stumbled upon this repo and this specific issue and the above exchange is such a great exchange discussing challenging issues with such a wonderful balance and appreciation of perspective from each side. I just have to say kudos to you both. I've recently been moving data around in DynamoDB and this explains a lot so thank you for that. |
I noticed 2 main issues with the batchWrite function.
Requests not throttled
When the amount of items is considerable the requests should be throttled to avoid issues; the throttling should be configurable by the caller or an exponential backoff could be implemented.
Consider a quite common scenario such as putting 100K items to a table; with the current batchWrite implementation this would result in 4000 promises running in parallel. With such a rate even a table configured as 'on-demand' will quickly expires the provisioned throughput.
Bad exception management
If an exception occurs during an await documentClient.batchWrite() call, and this will quickly happens when requests are not throttled, something really bad occurs: the await batchWrite() made by the caller does not wait anymore BUT the remaining chunks will still be written on the background!
It is important to note that, as stated in the official doc,
So if the throughput is not sufficient to write x out of 25 items no exception will be raised and the unprocessed items will be returned in the promise result, so far so good; but if 25 out of 25 items will not be written an exception will be raised.
The try...catch surrounding the return await Promise.all() will surely catch the exception but it is too late to keep the status of the writings under control.
My advice is to manage the exceptions inside the code of the promises and a list of all the failed requests should be returned to the caller.
To overcome these issues I made some little modifications which you can find in this gist.
I know that the real solution should be engineered better but I am a new comer in the javascript world so I do not really know what the best approach would be here.
Thanks for your work, I have learnt a lot studying your code.
The text was updated successfully, but these errors were encountered: