-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement materialized index scans by materializing lazy scans #1323
Implement materialized index scans by materializing lazy scans #1323
Conversation
… scan. This makes the code much simpler, and also shouldn't be to expensive, as the expensive parts (reading from disk and decompressing) are still done in parallel.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1323 +/- ##
==========================================
- Coverage 88.89% 88.89% -0.01%
==========================================
Files 326 326
Lines 28992 28931 -61
Branches 3214 3207 -7
==========================================
- Hits 25773 25718 -55
+ Misses 2066 2061 -5
+ Partials 1153 1152 -1 ☔ View full report in Codecov by Sentry. |
|
// block in parallel | ||
#pragma omp task | ||
{ | ||
if (!cancellationHandle->isCancelled()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the only usage of isCancelled()
outside of tests I think. I'd suggest removing the function or replacing it with something like ASSERT_THROW(throwIfCancelled())
where it makes sense to keep assertions around.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works like a charm and makes no difference for the first 20 example queries on Wikidata. See the revised description of the PR for more infos.
…eiburg#1323) This makes the code much simpler, and makes no difference for almost all queries. The expensive part (reading from disk and decompressing) is still done in parallel, only the writing to the `IdTable` is now serialized + there is an additional copy compared to before. An example query that is slower now because of this change is: materialize a large index scan (for example, for the predicate `rdf:type`) and group by subject (there is a shortcut for grouping by object when there are few objects). But such queries will become lazy soon anyway (see ad-freiburg#1350) and then this will be irrelevant.
This makes the code much simpler, and makes no difference for almost all queries. The expensive part (reading from disk and decompressing) is still done in parallel, only the writing to the
IdTable
is now serialized + there is an additional copy compared to before. An example query that is slower now because of this change is: materialize a large index scan (for example, for the predicaterdf:type
) and group by subject (there is a shortcut for grouping by object when there are few objects). But such queries will become lazy soon anyway (see #1350) and then this will be irrelevant.