-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Searching multiple pools with "from *" is much slower #5070
Comments
Repro is with Zed commit 38763f8. While I don't have the benefit of the user's specific data to test with, I've been able to reproduce a similar effect locally on my Macbook using the data from I first created separate pools to each hold a subset of the wrccdc data.
FWIW the Doing a "needle in a haystack" search for a value that I know is present in both pools, we get fast search performance on each individually.
However, doing the search across both pools at once with
And if I drop those pools and load the entire data set into one pool, it runs fast whether I target it directly or via
|
Thank you for the perfect text you wrote Even this problem exists during import and export Maximum hard drive reading speed of 120 MB from 2GB which increases the time of import and export |
Hello |
@SoftTools59654: No, there's not yet been a solution here. While we understand the performance hit is significant, it turns out to be a very difficult problem to solve. One of our developers has recently revisited it and is working on designing an approach. We'll update here when there's progress to report. |
Verified in Zed commit 6e61673. Using the same Macbook, data set, and query that I used above for the 11+ minute repro, now we see the same takes just over 15 seconds. While it's not guaranteed that the software will always be able to leverage all available CPU, in this case I can see that all my cores were near 100% busy during the query.
I'd not studied non-search operations like
I'd not baselined
This fix is available now for those willing to compile and run Thanks @mattnibs! |
When searching a text in a data about 100 GB and 80 GB
When using the from* command to read both files, the speed of reading data from the m2 hard drive is about 40MB. But when the same text is read without the from command, the speed of reading the file reaches about 2 GB. While the CPU is fully engaged, the file loading speed from the hard drive is very low
There is a lot of time difference in using the from command and searching one by one in the file. I tested this case several times and this case was there.
The text was updated successfully, but these errors were encountered: