-
Notifications
You must be signed in to change notification settings - Fork 288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speeding up inserts in blob storage #1243
Comments
Hi @aravindsrinivasan, thanks for trying out LanceDB. The methods you are using to insert all seem reasonable. I don't think you are using LanceDB in a way that would make it slow. I think the most productive thing you could do here is quantify how fast you are able to write on Azure with LanceDB vs some other library. For example, you can time how long it takes to insert 100,000 vectors. Then if you vectors are say 1024 dimensions, then you can estimate the data you wrote is about |
@wjones127 thank you for the response. Turns out my internet was the culprit -- plugging in the ethernet cable made it 10x faster. Generally speaking, what has your team found to be the fastest way to upload into an index? Parallelization didn't seem to work as well as sequential inserts with a large batch size. This feels counter intuitive to me so curious if this is to be expected |
Writing in batches is much more efficient in LanceDB. You could write batches of 10-100k in parallel, and that might work well. But writing <1k rows in parallel will perform poorly and produce a bad table layout, which would need to be fixed by calling compact_files. |
Thanks @wjones127. |
I'm finding that inserting data into a table is quite slow when it's backed by blob storage (Azure blob storage in my case). I've tried:
Table.add
) post table creationTable
object thread safe?)All of them seem to be equally slow. What's the best way to speed this up?
The text was updated successfully, but these errors were encountered: