Enable multiQuery optimization for has step

Fixes #3244

Signed-off-by: Oleksandr Porunov <alexandr.porunov@gmail.com>

docs/configs/janusgraph-cfg.md

@@ -347,10 +347,17 @@ Configuration options for query processing
 | Name | Description | Datatype | Default Value | Mutability |
 | ---- | ---- | ---- | ---- | ---- |
 | query.batch | Whether traversal queries should be batched when executed against the storage backend. This can lead to significant performance improvement if there is a non-trivial latency to the backend. | Boolean | true | MASKABLE |
-| query.batch-property-prefetch | Whether to do a batched pre-fetch of all properties on adjacent vertices against the storage backend prior to evaluating a has condition against those vertices. Because these vertex properties will be loaded into the transaction-level cache of recently-used vertices when the condition is evaluated this can lead to significant performance improvement if there are many edges to adjacent vertices and there is a non-trivial latency to the backend. | Boolean | false | MASKABLE |
 | query.fast-property | Whether to pre-fetch all properties on first singular vertex property access. This can eliminate backend calls on subsequent property access for the same vertex at the expense of retrieving all properties at once. This can be expensive for vertices with many properties | Boolean | true | MASKABLE |
 | query.force-index | Whether JanusGraph should throw an exception if a graph query cannot be answered using an index. Doing so limits the functionality of JanusGraph's graph queries but ensures that slow graph queries are avoided on large graphs. Recommended for production use of JanusGraph. | Boolean | false | MASKABLE |
 | query.hard-max-limit | If smart-limit is disabled and no limit is given in the query, query optimizer adds a limit in light of possibly large result sets. It works in the same way as smart-limit except that hard-max-limit is usually a large number. Default value is Integer.MAX_VALUE which effectively disables this behavior. This option does not take effect when smart-limit is enabled. | Integer | 2147483647 | MASKABLE |
+| query.has-step-batch-mode | Properties pre-fetching mode for `has` step. Used only when query.batch is enabled.<br>Supported modes:<br>- `all_properties` Pre-fetch all vertex properties on any property access<br>- `required_properties_only` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps<br>- `required_and_next_properties` Prefetch the same properties as with `required_properties_only` mode, but also prefetch
+properties which may be needed in the next properties access step like `values`, `properties,` `valueMap`, or `elementMap`.
+In case the next step is not one of those properties access steps then this mode behaves same as `required_properties_only`.
+In case the next step is one of the properties access steps with limited scope of properties, those properties will be
+pre-fetched together in the same multi-query.
+In case the next step is one of the properties access steps with unspecified scope of property keys then this mode
+behaves same as `all_properties`.<br>- `required_and_next_properties_or_all` Prefetch the same properties as with `required_properties_only`, but in case the next step is not
+`values`, `properties,` `valueMap`, or `elementMap` then acts like `all_properties`.<br>- `none` Skips `has` step batch properties pre-fetch optimization.<br> | String | required_and_next_properties | MASKABLE |
 | query.ignore-unknown-index-key | Whether to ignore undefined types encountered in user-provided index queries | Boolean | false | MASKABLE |
 | query.index-select-strategy | Name of the index selection strategy or full class name. Following shorthands can be used: <br>- `brute-force` (Try all combinations of index candidates and pick up optimal one)<br>- `approximate` (Use greedy algorithm to pick up approximately optimal index candidate)<br>- `threshold-based` (Use index-select-threshold to pick up either `approximate` or `threshold-based` strategy on runtime) | String | threshold-based | MASKABLE |
 | query.index-select-threshold | Threshold of deciding whether to use brute force enumeration algorithm or fast approximation algorithm for selecting suitable indexes. Selecting optimal indexes for a query is a NP-complete set cover problem. When number of suitable index candidates is no larger than threshold, JanusGraph uses brute force search with exponential time complexity to ensure the best combination of indexes is selected. Only effective when `threshold-based` index select strategy is chosen. | Integer | 10 | MASKABLE |

janusgraph-backend-testutils/src/main/java/org/janusgraph/testutil/JanusGraphAssert.java

@@ -201,4 +201,15 @@ public static Metrics getStepMetrics(TraversalMetrics traversalMetrics, Class<?
         }
         return null;
     }
+
+    public static Metrics getLastStepMetrics(TraversalMetrics traversalMetrics, Class<? extends Step> stepClass){
+        String stepMetricsName = stepClass.getSimpleName();
+        Metrics metricsToReturn = null;
+        for(Metrics metrics : traversalMetrics.getMetrics()){
+            if(metrics.getName().startsWith(stepMetricsName)){
+                metricsToReturn = metrics;
+            }
+        }
+        return metricsToReturn;
+    }
 }

janusgraph-core/src/main/java/org/janusgraph/core/TransactionBuilder.java

@@ -15,6 +15,7 @@
 package org.janusgraph.core;
 
 import org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode;
 
 import java.time.Instant;
 
@@ -147,6 +148,15 @@ public interface TransactionBuilder {
      */
     TransactionBuilder skipDBCacheRead();
 
+    /**
+     * Sets `has` step strategy mode.
+     * <p>
+     * Doesn't have any effect if multi-query was disabled via config `query.batch`.
+     *
+     * @return Object with the set `has` step strategy mode settings
+     */
+    TransactionBuilder setHasStepStrategyMode(MultiQueryHasStepStrategyMode hasStepStrategyMode);
+
     /**
      * Sets the group name for this transaction which provides a way for gathering
      * reporting on multiple transactions into one group.

janusgraph-core/src/main/java/org/janusgraph/graphdb/configuration/GraphDatabaseConfiguration.java

@@ -57,6 +57,7 @@
 import org.janusgraph.graphdb.query.index.BruteForceIndexSelectionStrategy;
 import org.janusgraph.graphdb.query.index.IndexSelectionStrategy;
 import org.janusgraph.graphdb.query.index.ThresholdBasedIndexSelectionStrategy;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.MultiQueryHasStepStrategyMode;
 import org.janusgraph.graphdb.transaction.StandardTransactionBuilder;
 import org.janusgraph.graphdb.types.system.ImplicitKey;
 import org.janusgraph.util.StringUtils;
@@ -299,11 +300,32 @@ public class GraphDatabaseConfiguration {
                 "will give the optimizer a chance to find more efficient execution plan but also increase the optimization overhead.",
             ConfigOption.Type.MASKABLE, true);
 
-    public static final ConfigOption<Boolean> BATCH_PROPERTY_PREFETCHING = new ConfigOption<>(QUERY_NS,"batch-property-prefetch",
-            "Whether to do a batched pre-fetch of all properties on adjacent vertices against the storage backend prior to evaluating a has condition against those vertices. " +
-                    "Because these vertex properties will be loaded into the transaction-level cache of recently-used vertices when the condition is evaluated this can " +
-                    "lead to significant performance improvement if there are many edges to adjacent vertices and there is a non-trivial latency to the backend.",
-            ConfigOption.Type.MASKABLE, false);
+    public static final ConfigOption<String> HAS_STEP_BATCH_MODE = new ConfigOption<>(QUERY_NS,"has-step-batch-mode",
+        String.format("Properties pre-fetching mode for `has` step. Used only when "+USE_MULTIQUERY.toStringWithoutRoot()+" is enabled.<br>" +
+                "Supported modes:<br>" +
+                "- `%s` Pre-fetch all vertex properties on any property access<br>" +
+                "- `%s` Pre-fetch necessary vertex properties for the whole chain of foldable `has` steps<br>" +
+                "- `%s` Prefetch the same properties as with `%s` mode, but also prefetch\n" +
+                "properties which may be needed in the next properties access step like `values`, `properties,` `valueMap`, or `elementMap`.\n" +
+                "In case the next step is not one of those properties access steps then this mode behaves same as `%s`.\n" +
+                "In case the next step is one of the properties access steps with limited scope of properties, those properties will be\n" +
+                "pre-fetched together in the same multi-query.\n" +
+                "In case the next step is one of the properties access steps with unspecified scope of property keys then this mode\n" +
+                "behaves same as `%s`.<br>"+
+                "- `%s` Prefetch the same properties as with `%s`, but in case the next step is not\n" +
+                "`values`, `properties,` `valueMap`, or `elementMap` then acts like `%s`.<br>"+
+                "- `%s` Skips `has` step batch properties pre-fetch optimization.<br>",
+            MultiQueryHasStepStrategyMode.ALL_PROPERTIES.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.REQUIRED_PROPERTIES_ONLY.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.REQUIRED_AND_NEXT_PROPERTIES.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.REQUIRED_PROPERTIES_ONLY.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.REQUIRED_PROPERTIES_ONLY.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.ALL_PROPERTIES.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.REQUIRED_AND_NEXT_PROPERTIES_OR_ALL.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.REQUIRED_PROPERTIES_ONLY.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.ALL_PROPERTIES.getConfigurationOptionName(),
+            MultiQueryHasStepStrategyMode.NONE.getConfigurationOptionName()),
+            ConfigOption.Type.MASKABLE, MultiQueryHasStepStrategyMode.REQUIRED_AND_NEXT_PROPERTIES.getConfigurationOptionName());
 
     // ################ SCHEMA #######################
     // ################################################
@@ -1268,13 +1290,12 @@ public boolean apply(@Nullable String s) {
     private boolean limitBatchSize;
     private boolean optimizerBackendAccess;
     private IndexSelectionStrategy indexSelectionStrategy;
-    private Boolean batchPropertyPrefetching;
     private boolean allowVertexIdSetting;
     private boolean allowCustomVertexIdType;
     private boolean logTransactions;
     private String metricsPrefix;
     private String unknownIndexKeyName;
-
+    private MultiQueryHasStepStrategyMode hasStepStrategyMode;
     private StoreFeatures storeFeatures = null;
 
     public GraphDatabaseConfiguration(ReadConfiguration configurationAtOpen, ModifiableConfiguration localConfiguration,
@@ -1380,8 +1401,8 @@ public IndexSelectionStrategy getIndexSelectionStrategy() {
         return indexSelectionStrategy;
     }
 
-    public boolean batchPropertyPrefetching() {
-        return batchPropertyPrefetching;
+    public MultiQueryHasStepStrategyMode hasStepStrategyMode() {
+        return hasStepStrategyMode;
     }
 
     public boolean adjustQueryLimit() {
@@ -1512,7 +1533,9 @@ private void preLoadConfiguration() {
         indexSelectionStrategy = Backend.getImplementationClass(configuration, configuration.get(INDEX_SELECT_STRATEGY),
             REGISTERED_INDEX_SELECTION_STRATEGIES);
         optimizerBackendAccess = configuration.get(OPTIMIZER_BACKEND_ACCESS);
-        batchPropertyPrefetching = configuration.get(BATCH_PROPERTY_PREFETCHING);
+        String hasStepBatchModeName = configuration.get(HAS_STEP_BATCH_MODE);
+        hasStepStrategyMode = Arrays.stream(MultiQueryHasStepStrategyMode.values()).filter(mode -> mode.getConfigurationOptionName().equals(hasStepBatchModeName)).findAny().orElse(null);
+        Preconditions.checkNotNull(hasStepStrategyMode, hasStepBatchModeName+" is not recognized as one of the supported values of `"+HAS_STEP_BATCH_MODE.toStringWithoutRoot()+"`");
         adjustQueryLimit = configuration.get(ADJUST_LIMIT);
         hardMaxLimit = configuration.get(HARD_MAX_LIMIT);
 

janusgraph-core/src/main/java/org/janusgraph/graphdb/database/StandardJanusGraph.java

@@ -65,6 +65,7 @@
 import org.janusgraph.graphdb.database.cache.SchemaCache;
 import org.janusgraph.graphdb.database.idassigner.VertexIDAssigner;
 import org.janusgraph.graphdb.database.idhandling.IDHandler;
+import org.janusgraph.graphdb.tinkerpop.optimize.strategy.JanusGraphHasStepStrategy;
 import org.janusgraph.util.IDUtils;
 import org.janusgraph.graphdb.database.index.IndexInfoRetriever;
 import org.janusgraph.graphdb.database.index.IndexUpdate;
@@ -149,6 +150,7 @@ public class StandardJanusGraph extends JanusGraphBlueprintsGraph {
                                JanusGraphMixedIndexAggStrategy.instance(),
                                JanusGraphMixedIndexCountStrategy.instance(),
                                JanusGraphStepStrategy.instance(),
+                               JanusGraphHasStepStrategy.instance(),
                                JanusGraphIoRegistrationStrategy.instance());
 
         //Register with cache

janusgraph-core/src/main/java/org/janusgraph/graphdb/query/vertex/BasicVertexCentricQueryUtil.java

@@ -0,0 +1,36 @@
+// Copyright 2023 JanusGraph Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package org.janusgraph.graphdb.query.vertex;
+
+import org.apache.tinkerpop.gremlin.structure.Direction;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.janusgraph.core.VertexLabel;
+import org.janusgraph.graphdb.types.VertexLabelVertex;
+import org.janusgraph.graphdb.types.system.BaseLabel;
+import org.janusgraph.graphdb.types.system.BaseVertexLabel;
+
+public class BasicVertexCentricQueryUtil {
+
+    public static <Q extends BasicVertexCentricQueryBuilder<? extends BasicVertexCentricQueryBuilder<?>>> Q withLabelVertices(Q queryBuilder){
+        queryBuilder.noPartitionRestriction().type(BaseLabel.VertexLabelEdge).direction(Direction.OUT);
+        return queryBuilder;
+    }
+
+    public static VertexLabel castToVertexLabel(Vertex vertexLabel){
+        if (vertexLabel==null) return BaseVertexLabel.DEFAULT_VERTEXLABEL;
+        else return (VertexLabelVertex)vertexLabel;
+    }
+
+}

janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/JanusGraphTraversalUtil.java

@@ -122,6 +122,13 @@ public static Step getNextNonIdentityStep(final Step start) {
         return currentStep;
     }
 
+    public static Step getPreviousNonIdentityStep(final Step start) {
+        Step currentStep = start.getPreviousStep();
+        //Skip over identity steps
+        while (currentStep instanceof IdentityStep) currentStep = currentStep.getPreviousStep();
+        return currentStep;
+    }
+
     public static JanusGraphTransaction getTx(Traversal.Admin<?, ?> traversal) {
         final JanusGraphTransaction tx;
         Optional<Graph> optGraph = TraversalHelper.getRootTraversal(traversal.asAdmin()).getGraph();