-
-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using overlapped IO for batch insert and query #16
Comments
Open in different threads different transactions and query the same table. It works out of the box |
This is not exactly the same as overlapped IO. You can do several overlapped IO without having to spawn more threads. Also I noticed big perf hit if I make one transaction per query. |
Of course, you create several transactions to execute in parallel fetching batches, not one select. |
Thanks, I'll try that. I receive one big batch to fetch, so indeed I can split it in smaller batch to several worker threads. Might be good enough for my case, just that I think it is even more efficient to use BeginRead/BeginEnd on the file system, as it would not require several threads. I let you know how it goes, thanks. |
An example: Here https://github.com/hhblaze/DBreeze/blob/master/DBreeze/Storage/FSR.cs#L833 you are looping and doing many write sequentially. I will try to do that and benchmark a bit, I have some crappy hard drive which make measuring that easy. |
Forget what I said, you can't call several BeginRead without reading results sequentially. https://msdn.microsoft.com/en-us/library/zxt5ahzw(v=vs.110).aspx |
At least for queries, there is no way to fetch data in batch efficiently by using overlapped IO.
This is needed for several reasons for doing that:
The text was updated successfully, but these errors were encountered: