Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve type inference performance to enable large queries #6431

Merged
merged 50 commits into from
Sep 16, 2021

Conversation

flyingsilverfin
Copy link
Member

@flyingsilverfin flyingsilverfin commented Sep 2, 2021

What is the goal of this PR?

Enable large queries to be processed by our type inference engine. Previously, large queries would cause the query to stall. A new type inference algorithm is implemented to massively speed up static type checking that is performed on every query.

What are the changes implemented in this PR?

  • Create GraphTraversal.Unrestricted (previously just GraphTraversal) and GraphTraversal.Type, which is a graph traversal that can only operate over the schema
  • Add a basic, preliminary type inference step based just on the constraints, which is performed when building the type resolver traversal
  • Rename Procedure to PermutationProcedure, and create TypeCombinationProcedure
  • Create TypeCombinationGetter, which executes a TypeCombinationProcedure and returns an Optional with the type combination that was found
  • Complete the GraphProcedure.Builder (intermediate state utilised it - we do not use it anymore)
  • clean up TraversalEngine API

@@ -314,7 +316,7 @@ public void setSupertype(AttributeType superType) {

@Override
public FunctionalIterator<AttributeTypeImpl> getSubtypes() {
return getSubtypes(v -> {
return graphMgr.schema().getSubtypes(vertex).map(v -> {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we move the method of getting subtypes from TypeImpl to TypeGraph so that the type graph doesn't have to repeat the exact code to get subtypes

@@ -141,10 +141,6 @@ private void validateTypeHierarchyIsNotCyclic() {
}
}

<TYPE extends Type> FunctionalIterator<TYPE> getSubtypes(Function<TypeVertex, TYPE> typeConstructor) {
return tree(vertex, v -> v.ins().edge(SUB).from()).map(typeConstructor);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

functionality moved to TypeGraph

Comment on lines 125 to 136
private Set<Label> relations;
private Set<Label> roles;
private Set<Label> rolePlayers;
private Set<Label> rolesPlayed;
private Set<Label> attributes;
private Set<Label> attributesOwned;
private Set<Label> attributeOwners;
private Set<Label> stringAttributes;
private Set<Label> longAttributes;
private Set<Label> doubleAttributes;
private Set<Label> booleanAttributes;
private Set<Label> datetimeAttributes;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

caches of sets of type labels we require for more efficient type resolution. We compute these over and over and they must be bound to the lifetime of the TypeGraph - so we reuse the Cache object here to store them.

import java.util.Set;

public class LogicCache {

private CommonCache<Structure, Map<Identifier.Variable.Retrievable, Set<Label>>> typeResolverCache;
private CommonCache<Structure, Optional<Map<Identifier.Variable.Retrievable, Set<Label>>>> typeResolverCache;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use an optional instead of an empty map as a signal of an unsatisfiable query - this is more logical

}

public FunctionalIterator<Map<Identifier.Variable.Name, Label>> namedCombinations(Conjunction conjunction, boolean insertable) {
GraphTraversal resolverTraversal = new GraphTraversal();
TraversalBuilder traversalBuilder = new TraversalBuilder(conjunction, conceptMgr, resolverTraversal, 0, insertable);
TypeTraversal resolverTraversal = new TypeTraversal();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we created a new type of traversal just for traversing the type graph, called TypeTraversal, which we use here

Comment on lines 63 to 58
public FunctionalIterator<VertexMap> iterator(GraphTraversal traversal) {
return iterator(traversal, false);
traversal.initialise(cache);
return traversal.iterator(graphMgr);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

general Graph traversals must be query planned and cache the plan

else traversal.initialise(cache);
return traversal.iterator(graphMgr, singleUse);
public FunctionalIterator<VertexMap> iterator(TypeTraversal traversal) {
return traversal.iterator(graphMgr);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type traversal plans are NOT cached

Comment on lines 81 to 84
public Optional<Map<Retrievable, Set<TypeVertex>>> combination(TypeTraversal traversal,
Set<Retrievable> abstractDisallowed) {
return new TypeCombination(graphMgr, TypeCombinationProcedure.of(traversal), traversal.parameters(),
traversal.filter(), abstractDisallowed).get();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and TypeTraversals can also be executed as a Combination, which presents an Optional API instead of an Iterator

@@ -60,6 +68,38 @@ public Parameters parameters() {
return parameters;
}

abstract FunctionalIterator<VertexMap> iterator(GraphManager graphMgr);

FunctionalIterator<VertexMap> iterator(GraphManager graphMgr, List<Planner> planners, boolean singleUse, Set<Identifier.Variable.Retrievable> filter) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a common implementation for iterating over things that have Planner objects - this is used by both GraphTraversal and TypeTraversal

traversal/graph/TraversalVertex.java Outdated Show resolved Hide resolved
@flyingsilverfin flyingsilverfin marked this pull request as ready for review September 3, 2021 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants