All changes needed for the Evaluation paper #355
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-- now we have completely removed the dependency from google::sparsehash and migrated to absl.
-- Note that so far, no unescaping is performed.
- Transitive Paths use HashSets which now also have undefined orderings with absl. Tests don't rely on this anymore - The case, where a prefix was used with an empty "content" (e.g. <a> wd: <b>) was broken before, luckily there was a unit test and this is now fixed.
…TL file. - Previously there was a "parsing of ttl has failed, but there is still content left" warning, although the remainder of the ttl input was only whitespace. - This was due to a bug in the Parser's skipWhitespace() function which failed if the input consisted of ONLY whitespace. This is now fixed.
…ashSet # Conflicts: # src/parser/TurtleParser.cpp # src/parser/TurtleParser.h
- also apply the normalization of literals correctly during index build time - Adapt the Index unit tests to "legal" knowledge bases - Disable the CTRE parser for now, since it becomes awefully slow with the PnameNS and PnLocal changes for some reasons. - TODO: Maybe we want to renable the CTRE Parser with the old "wrong" behavior as a very fast way to parse Wikidata
- The previous version put all triples and filters that appeared directly in a WHERE clause together.
- this is not correct, since always when an OPTIONAL, UNION, SUBQUERY etc. appears a new scope
is started. For example in
SELECT ?x ?y WHERE {
?x <is-a> <Scientist>
OPTIONAL { ?x <Spouse> ?y}
FILTER ?x < <Ada_Lovelace>
}
the filter should have no effect, because it is not in the same scope as the x is a scientist triple.
the probably more useful query would be
SELECT ?x ?y WHERE {
?x <is-a> <Scientist> .
FILTER ?x < <Ada_Lovelace>
OPTIONAL { ?x <Spouse> ?y}
}
where the filter is actually applied on the entities that are scientists.
However, QLever previously handled these queries both in the second way.
- To correct this, implemented a BasicGraphPattern (triples, filters and values that are "directly" in the Body of another graph pattern or a WHERE clause) class.
BasicGraphPatterns are also children of their parent body.
- In addition, Optionals were also treated wrong as the GraphPatterns must be treated in order which makes a difference for Optionals.
- This version of the QueryPlanner does some degree of optimization, however one could optimize multiple adjacent GraphPatterns that are not Optional, if their
scopes are respected.
- TODO: There still are bugs in the handling of optionals as the SPARQL standard knows "unbound values" which can be rebound, e.g.
SELECT ?x ?name WHERE {
?x <is-a> <human>
OPTIONAL {?x <TwitterAccount> ?name}
OPTIONAL {?x rdfs:label ?name }
}
which should only add the names only for humans that don't have twitter accounts and the handling of the empty query
SELECT ?x WHERE {} (1 result where ?x is unbound/NULL)
- TODO: Some candidate plans found by optimizing a TransitivePath are currently unable to be joined with adjacent patterns.
It seems that we can always find a suitable candidate, so that this works, but we still should inspect, whether this is the correct behavior.
…type of GraphPatternOperation. The parser already compiles.
TODO: clean up, self-review, commit messages and squash/rewrite history.
…identical again. Next step: make merge call join.
TODO<Hannah> checkout how this performs on the Wikidata evaluation.
TODO: squash, git history and then self-review (but first let them do the ParsedQuery stuff.)
…. Not yet working at all.
…ern trick in cases where it was actually legal.
TODO: merge with the new optimizer to properly include it, and make this all work and test it.
# Conflicts: # src/parser/ParsedQuery.h
f compile. Not yet done at all.
…rVariable). - There also is a stub for BIND(<something> + ?else as ?sum), but that is not yet fully integrated yet.
- No more doubled application of Filters in case of BIND() clauses - now the parser accepts the optional '.' after a bind clause.
- It keeps track of the number of maximally allowed bytes, using a synchronized, shared state, checks if an allocation is OK. - The actual allocation is still done by a std::allocator. - The IdTable now also internally uses an allocator template insteead of malloc and free.
- Currently only working in tests - seem to have found a compiler bug in g++ 7.5
Required minor adaptations for a lot of Unit tests. TODO: test, if we can indeed avoid the bad_allocs on galera. TODO: more detailed unit tests.
…put. TODO: UnitTest this.
TODO: limit the number of threads and make it configurable.
…o far computed operations in the response
This greatly increases the time needed to start QLever - The different steps (Handling the prefix Compression and the escaping) can be trivially parallelized
-The HashMaps in the Transitive Path operation previously contained one unnecessary level of indirection which is removed by this commit. - This increases the performance of the transitive paths.
Reason was an invalid dereferencing of an end-iterator.
…to pr328+pr330+pr332+pr342+pr34 # Conflicts: # src/engine/CancelableSort/SortExampleMain.cpp # src/engine/RuntimeInformation.h # src/engine/TransitivePath.cpp # src/index/Index.cpp # src/index/PrefixHeuristic.h # src/index/StringSortComparator.h # src/index/Vocabulary.h # src/index/VocabularyGeneratorImpl.h
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Everything merged together which is also in single PRs and needed for the autcompletion to work.
DO NOT MERGE AS IS