Bidirectional expand into #1047

jeffreylovitz · 2020-04-01T20:18:38Z

Resolves #1036, introduces shared logic for populating referenced edges in CondTraverse and ExpandInto operations.

DvirDukhan · 2020-04-01T21:06:09Z

tests/flow/test_bidirectional_traversals.py

+        query = """MATCH (a), (b) WITH a, b MATCH (a)-[e:E]-(b) RETURN ID(e), a.val, b.val ORDER BY ID(e), a.val, b.val"""
+        actual_result = acyclic_graph.query(query)
+        expected_result = [[0, 'v1', 'v2'],
+                           [0, 'v2', 'v1'],
+                           [1, 'v2', 'v3'],
+                           [1, 'v3', 'v2']]
+        self.env.assertEquals(actual_result.result_set, expected_result)
+
+        # Verify result against the equivalent conditional traversal.
+        query = """MATCH (a)-[e:E]-(b) RETURN ID(e), a.val, b.val ORDER BY ID(e), a.val, b.val"""


set id as edge property as we can't rely on the order of the redisgraph-py flushes the data

DvirDukhan · 2020-04-01T21:10:34Z

src/execution_plan/ops/shared/traverse_functions.h

+#include "../../execution_plan.h"
+#include "../../../arithmetic/algebraic_expression.h"
+
+typedef struct {


Please add a comment where this struct is used

DvirDukhan · 2020-04-01T21:12:51Z

src/execution_plan/ops/shared/traverse_functions.h

+// Collect all appropriate edges between the given endpoints.
+void Traverse_CollectEdges(EdgeTraverseData *edge_data, NodeID src, NodeID dest);
+
+// If a matching edge is available, pop it and add it to the Record.


// Removes an available matching edge from edges array and sets it in the Record.

DvirDukhan · 2020-04-01T21:17:41Z

tests/tck/features/MatchingSelfRelationships.feature

@@ -63,6 +63,7 @@ Feature: MatchingSelfRelationships
      | 1        |
    And no side effects

+@skip


did it pass by accident?

Yes. Bidirectional traversals typically return each edge twice (incoming and outgoing), but self-loops in Cypher still should only return once - I'm guessing this is a consequence of the edge uniqueness constraint we don't currently adhere to. I conferred with Roi and we agreed that these should be skips for the moment.

tests/tck/features/MatchingSelfRelationships.feature

DvirDukhan · 2020-04-01T21:24:43Z

src/execution_plan/ops/shared/traverse_functions.h

+} EdgeTraverseData;
+
+// Initialize an EdgeTraverseData struct to populate edges appropriately for traversal operations.
+void Traverse_NewEdgeData(EdgeTraverseData *edge_data, AlgebraicExpression *ae,


I think that returning an allocated struct is the way to go here.

DvirDukhan · 2020-04-01T21:27:15Z

src/execution_plan/ops/op_expand_into.c

 	if(op->ae) {
 		AlgebraicExpression_Free(op->ae);
 		op->ae = NULL;
 	}

+	if(op->setEdge) Traverse_FreeEdgeData(&op->edge_data);


I think that it is better to have a pointer to an allocated struct.
Call the free function regardless and remove line 189 as it is breaking abstraction

DvirDukhan · 2020-04-01T21:27:31Z

src/execution_plan/ops/op_conditional_traverse.h

-	bool setEdge;               // Edge needs to be set.
-	Edge *edges;                // Discovered edges.
+	bool setEdge;               // Edge needs to be set in the Record.
+	EdgeTraverseData edge_data; // Edge collection data if the edge needs to be set.


I think that it is better to have a pointer to an allocated struct.

DvirDukhan · 2020-04-01T21:28:24Z

src/execution_plan/ops/shared/traverse_functions.h

+	Edge *edges;                // Flexible array of all matching edges for the current endpoints.
+	int edgeIdx;                // The Record index for the referenced edge.
+	GRAPH_EDGE_DIR direction;   // The direction of the referenced edge being traversed.
+} EdgeTraverseData;


Maybe TraverseEdgeContext is more suitable?

👍 That's better, thanks!

DvirDukhan · 2020-04-01T21:32:15Z

src/execution_plan/ops/op_conditional_traverse.c

-		array_free(op->edgeRelationTypes);
-		op->edgeRelationTypes = NULL;
-	}
+	if(op->setEdge) Traverse_FreeEdgeData(&op->edge_data);


please call Free regardless, remove the array_clear from line 177 as it is breakinging abstraction
use a pointer to an allocated struct

DvirDukhan · 2020-04-02T17:32:44Z

src/execution_plan/ops/shared/traverse_functions.c

+void Traverse_FreeEdgeCtx(EdgeTraverseCtx *edge_ctx) {
+	if(!edge_ctx) return;
+
+	array_free(edge_ctx->edges);


maybe I missed it. Do we gurentee for the array to be empty? Do we need to free the edges?

The Edges themselves aren't heap allocations, they're contiguous structs in this array: https://github.com/RedisGraph/RedisGraph/blob/master/src/graph/graph.c#L204-L206

swilly22

A few simple comments.

swilly22 · 2020-05-13T05:53:00Z

src/execution_plan/ops/shared/traverse_functions.h

+typedef struct {
+	int *edgeRelationTypes;     // The relation type IDs that should be collected.
+	Edge *edges;                // Flexible array of all matching edges for the current endpoints.
+	int edgeIdx;                // The Record index for the referenced edge.


I think edgeRecIdx is a bit clearer.

swilly22 · 2020-05-13T05:56:56Z

src/execution_plan/ops/shared/traverse_functions.c

+	if(e->bidirectional) {
+		// Bidirectional edges matching incoming and outgoing edges.
+		edge_ctx->direction = GRAPH_EDGE_DIR_BOTH;
+	} else if(AlgebraicExpression_ContainsOp(ae, AL_EXP_TRANSPOSE)) {
+		/* If this operation traverses a transposed edge, the source and destination nodes
+		 * will be swapped in the Record. */
+		edge_ctx->direction = GRAPH_EDGE_DIR_INCOMING;
+	} else {
+		// The default traversal direction is outgoing.
+		edge_ctx->direction = GRAPH_EDGE_DIR_OUTGOING;
+	}


Might consider moving this logic into a separate local static function

swilly22 · 2020-05-13T05:58:49Z

src/execution_plan/ops/shared/traverse_functions.c

+	return edge_ctx;
+}
+
+// Collect edges between the source and destination nodes matching the op's traversal direction.


Comment should mention that this function modifies/populates edge_ctx edges array.

swilly22 · 2020-05-13T06:13:42Z

src/execution_plan/ops/op_expand_into.h

@@ -7,6 +7,7 @@
 #pragma once

 #include "op.h"
+#include "shared/traverse_functions.h"
 #include "../execution_plan.h"
 #include "../../graph/graph.h"
 #include "../../graph/entities/edge.h"


Please go over this include list, see if all are necessary.

swilly22 · 2020-05-13T06:16:05Z

src/execution_plan/ops/op_expand_into.c

 	op->graph = g;
 	op->ae = ae;
 	op->r = NULL;
-	op->edges = NULL;
 	op->F = GrB_NULL;
 	op->M = GrB_NULL;
 	op->recordCount = 0;
-	op->edgeRelationTypes = NULL;
-	op->recordsCap = 0;
 	op->records = NULL;
+	op->recordsCap = 0;


It feels a waste to use rm_calloc but then set explicit NULLs,
if we still want to do this, we should set edge_ctx to NULL at this point.

swilly22 · 2020-05-13T06:28:07Z

src/execution_plan/ops/op_conditional_traverse.c

-		__CondTraverse_CollectEdges(op, op->destNodeIdx, op->srcNodeIdx);
-		return;
-	}
+static inline int CondTraverseToString(const OpBase *ctx, char *buf, uint buf_len) {


Although this function is basically a proxy function, it doesn't call too often for it to be declared as inline

swilly22 · 2020-05-13T06:28:57Z

src/execution_plan/ops/op_conditional_traverse.c

 	CondTraverse *op = rm_calloc(1, sizeof(CondTraverse));
 	op->graph = g;
 	op->ae = ae;
 	op->r = NULL;
 	op->iter = NULL;
-	op->edges = NULL;
 	op->F = GrB_NULL;
 	op->M = GrB_NULL;
-	op->recordsLen = 0;
-	op->direction = GRAPH_EDGE_DIR_OUTGOING;
-	op->edgeRelationTypes = NULL;
+	op->recordCount = 0;


See comment on NewExpandIntoOp

* Support float inputs for modulo computations (#895) * Add references * Update mkdocs.yml * Update References.md * fixed ast mapping for path filter (#896) * fixed ast mapping for path filter * renamed tests. open a new redis graph client per tests Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added mutex per matrix (#898) * added mutex per matrix * fixed PR comments * updated license headers to 2020 (#902) * Preserve order of op's children array when introducing index scans (#912) * Fix memory leaks (#917) * Fix memory leaks * Use original logic for MarkWriter * added filter tree clone (#915) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Optimize cp (#906) * added optimize cp * fixed a bug. added test * add optimization_util files * fixed PR comments * fixed PR comments * added multiple branch cp optimization * fixed PR comment * added in place replacement at ExecutionPlan_RemoveOp * added test for cp optimization and semi apply * fixed PR comments * Added non existing entity runtime error (#919) * Added non existing entity runtime error * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * RediSearch query error reporting (#925) * added filter tree compaction (#922) * added filter tree compaction * fixed PR comments * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Handle both possible execution orders in concurrent rename test (#926) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Refferencing a RedisGraph javascript client library (#928) * Improve client libraries sections on both readme and clients.md Clients libraries are subject to repetitive edition, it seems better to have a concise formatting * refferencing my own contribution to clients libraries Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Rust client (#931) * Update README.md * Update clients.md * intoduced fpClone in OpBase (#930) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Apply deplete match stream (#913) * Deplete Apply op's match stream for every left-hand Record * Disallow ExpandInto ops on variables beneath Apply ops * Add tests * PR fixes * Argument op holds one Record * PR fixes * Improve logic for building OpArgument modifies arrays * standardize logic for building tmp ExecutionPlans * Remove sub-ExecutionPlan logic * Fix CondTraverseReset routine * Fix ExpandIntoReset * Fix variable-length QueryGraph pointer * Always reset match branch * Add explanatory comment * PR fixes * PR fixes * raxValues returns void pointer array Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * increased allowed parameters count to UNIT_MAX. fixed a bug in query_ctx (#933) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * introduce mapping between record entries and projected columns (#936) * added arr clone with cb (#937) * added arr clone with cb * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Leak fixes (#935) * Close Redis key handles * Fix memory leak in checking whether procedures are read-only * Improve explanatory comment Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * label matrix should be fetch right before eval (#938) * More sensible function signature for SIValue_Free (#941) * Resolve memory leaks on Path SIValues (#940) * Resolve memory leaks on Path SIValues * Update op_delete.c * Update op_cond_var_len_traverse.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * GraphBLAS 3.2.0 (#942) * GraphBLAS 3.2.0 * updated makefiles * Remove unnecessary GraphBLAS build flag Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Simplify OpAggregate logic, remove unnecessary struct members (#947) * Simplify OpAggregate logic, remove unnecessary struct members * Update op_aggregate.c * Update op_aggregate.c * Update op_aggregate.c * Update op_aggregate.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add compile-time error for unsupported AST node types (#944) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * removed unused operations and types (#950) * remove GraphBLAS cacheing from CI (#953) * switch from OR AND semiring to ANY PAIR (#955) * using the structured semiring we can combine relation and relation ma… (#964) * using the structured semiring we can combine relation and relation mapping into a single matrix * address PR comments * Update graph.c * decoupled result set from execution plan (#929) * decoupled result set from execution plan * after rebase merge * fixed PR comments * Restored resulte set * fixed PR comments * fixed PR comments * fixed PR comments * Update ast.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Always free emptied space on AlgebraicExpression replacement (#986) * Fix leak on full-text index queries with syntax error (#985) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix leak on index scan IN optimization (#984) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Minor updates to bulk deletion (#994) * Remove redundant clone from bulk insertion of string props (#993) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix memory leaks on RDB-loaded strings (#991) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add flow test to validate (#967) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Update outdated logic for aggregate groups (#968) * Update outdated logic for aggregate groups * Update group.c * Update op_aggregate.c * Update op_aggregate.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added all_node_scan clone (#960) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added op argument clone (#959) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added cartesian product clone (#963) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added expand into clone (#973) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added project op clone (#980) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added skip op clone (#983) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added op unwind clone (#988) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added limit op clone (#976) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added proc call clone (#979) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added filter op clone (#974) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added delete op clone (#970) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added distinct op clone (#972) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add hash join op clone (#989) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added semi apply clone (#982) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added results op clone (#981) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added join op clone (#975) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added apply multiplexer clone (#961) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added cond var len traverse clone (#966) * added cond var len traverse clone * added conditional traverse clone * fixed PR comments * Reintroduce logic for freeing memory on run-time errors (#992) * Reintroduce logic for freeing memory on run-time errors * Remove VolatileRecord logic for freeing after run-time errors * Fix memory leaks on run-time errors in OpProject Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Free internal edge arrays on relationship matrix deletion (#997) * Free internal edge arrays on relationship matrix deletion * Update graph.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add debug function to print query (#995) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added clone for create, merge, merge_create, update (#969) * added clone for create, merge, merge_create, update * Update ast_shared.c * Update ast_shared.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Agg func fixes (#946) * Fix memory leaks in Collect function * Fix leak in aggregate func's SIValue result * Fix leak in children of aggregate function call * Simplify variable-length path free logic * Improve ownership logic in Collect, add explanatory comments * PR fix Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added label scan clone (#978) * added label scan clone * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added id seek clone (#977) * added id seek clone * added better comment on the range clone * Update op_node_by_id_seek.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added op aggreage clone (#958) * added op aggreage clone * fixed PR comments * Update op_aggregate.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add install instructions for OpenMP (#1006) * Add install instructions for OpenMP * Update README * added op sort clone (#987) * added op sort clone * fixed PR comments * removed free_list logic * Update execution_plan.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * re-enable graph.profile (#1004) * Re-enable GraphBLAS circle-ci cache (#1007) * Fix leak on projected heap-allocated graph entities (#996) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * GRAPH.SLOWLOG (#897) * slowlog WIP * slowlog per graph * updated docs * Add longer-running query to slowlog flow test * avoid race, log only GRAPH.QUERY Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Use NOP label scan if range iterator construction fails (#1001) * Use NOP label scan if range iterator construction fails * deplete iterator for invalid range * dont access op consume function directly Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * removed redisearch optimization (#1010) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Note ReplicaOf memory constraint in Rampfile (#1011) * Automate leak check (#1002) * Add make memcheck rule * Add Circle memcheck job * Don't use Docker image for automated leak checking * Enable log names by migrating flow test Env initialization * Disable invalid TCK scenarios of issue #945 * Run memory test without optimizations after packaging artifacts * Add suppressions for leaks in flush-vs-shutdown race Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Write longer-running query for slowlog test (#1012) * OPType bitmask switched to contiguous enum (#1013) * OPType bitmask switched to contiguous enum * PR fixes * Clean up op tree modification code * Add static array of all scan ops * Remove unnecessary conditional * Remove static type array for unoptimized scans * Validate arity per-command rather than globally (#1023) * Ar exp param (#990) * wip * wip * after rebase * updated libcypher parser * added AR_EXP_PARAM * fixed PR comments * fixed PR comments * fixed PR comments * fixed PR comments * Update arithmetic_expression.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix invalid op sequence when introducing index scans (#1028) * Fix invalid op sequence when partially replacing filters with index scans * Simplify op-freeing logic in utilizeIndices * Automate testing against enterprise v5.4.14 * push down transpose operations (#1032) * push down transpose operations * added test for transposed bi-directional edge * Update algebraic_expression_optimization.c * allow skip and limit to be parametes. (#1020) * wip * test pass * added skip limit params test * added failure test * fixed PR comments * fixed online review comments * restored traverse record cap * fixing memory leaks * fixed PR comments * changed redisearch version to 1.6.11 (#1035) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added datablock out of order ops (#1022) * added datablock out of order ops * fixed PR comments * Update oo_datablock.h * Update oo_datablock.c * fixed PR comments * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Adds versioned documentation (#1034) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * deploy on package release only (#1049) * deploy on package release only * fixed yml * changed workflows * changed workflows * fixed review comments * Optional match (#1043) * Enable TCK tests * Introduce Optional and Apply ops * Modify mock AST logic * Emit error on queries beginning with OPTIONAL MATCH * Return null on property accesses of null graph entity * Disallow OPTIONAL MATCH...MATCH queries * Fix OPTIONAL filter placement * Enable TCK tests * NULL handling for path functions * NULL handling for GraphEntity and list functions * WIP improve mock AST logic * Add flow tests * Improve AST mock logic * Error handling for SET and CREATE on null entities * Record_Get refactor * Test null handling * Minor cleanup * Add documentation * Simplify toPath null handling * Improve comments * Allow OPTIONAL MATCH as first clause * Simplify null-checking logic in create ops * Use branch of Python client for testing * PR fixes * PR fixes * Remove Record_GetScalar interface * PR fixes * PR fixes * Add demo query for OPTIONAL MATCH * Use standard Python client for automation * Emit all columns as SIValues in compact formatter * Improve flow test for null entities in first result Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added named path example (#1048) * added named path example * fixed pr comments * Better compile-time checking for undefined variables (#1063) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures * Add path comparison, streamline hashing logic (#1056) * Value comparison call correctly compares paths * Fix bug in DISTINCT paths and arrays * Fix memory leak in ValueHashJoin * Remove conditionals from Record hashing logic * Improve test coverage * Simplify memory management in ValueHashJoin Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Update README.md * Update index.md * Update README.md forum (#1073) * Update README.md forum * ReduceScan and ReduceTraversal respect variable scopes (#1070) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) * Guarantee that NOT conditions are unary (#1092) * is -> ID in docs (#1090) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Alpine variant of Dockerfile (#1087) * Calling autogen.sh fails if execute permissions are not set. Rather than do that, explicitly use "sh" to run it. * Explicitly include sys/types.h Several types (u_intN_t) are defined in this header, and on some systems not included indirectly. Therefore include the file directly. * Add an Alpine variant of the Dockerfile. * Change alpine image basis to Redis v6. Tag the image as alpine (without redis version). Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Redis 6 in builds (#1060) * changed dokerfile and ci * New installation scheme in Dockerfile * fixes 1 * Review fixes 1 Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Rafi Einstein <rafi@redislabs.com> * Fix for RAMP failure (#1093) * Graph encoding v7 - support replica-of (#1054) * added entities threshold to config * wip * added commit flow, without meta type * new graphmeta type * wip * added decode v6 * added new encoding flow * fixed edge encoding * done decoding * added entities threshold to config * wip * added commit flow, without meta type * wip * added decode v6 * added new encoding flow * fixed edge encoding * added tags * wip * wip * tests pass * wip * moved to redis 6, solved flushall. moved to uuid keys. adeded decode context * wip * moved to redis 6 events to handle keyspace * added graph pending replication * wip * wip * changed tagging * wip * tested for redis6 and redis5 * fixed ubuntu build error. added comments * changed get redis major version location * pr comments. wip * refactor. wip * fixed PR comments * fixed memeory leak * added delete guards * PR comments, wip * removed meta context type. removed uuid from meta key name * fixed edges array encoding. added tests * added skip test for redis 5 * Simplify meta key construction * fixed PR comments * added aux fields * added module replicating error. WIP * added v4, v6 rdb decode test * fixed v4 decode memory leak * fixed PR comments * fixe PR comments * fixed PR comments * fixed PR comments * fixed PR comments * added tear down to flow test to avoid race between RLTest and RG * Fix memory leak in v4 deserialization * huge refactor: single encoder/decoder logic. created config object. create redis-server version object. May support redis 5 * wip * changed virtual keys encoding logic. wip * tested on redis5. logs added * added uuid to meta keys * removed server version checking in flow tests * changed ramp * fixed redis server validation * fixed PR comments * fixed PR comments Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Update op_node_by_id_seek.c Fix, wrong variable assignment. * Update value.c removed.boolean switch * Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) * Fixed compiler typo (#1097) Paragraph on OSX build should read "CXX" instead of "CPP". * Union bugfixes (#1052) * Improve scoping rules in validating UNION queries * Bugfix in uniquing column containing both nodes and edges * Add flow tests * PR fixes * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments * throw runtime-error on missing query parameters (#1100) * removed query parameters annotations (#1101) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) * changed RediSearch version to 1.8 (#1103) * changed RediSearch version to 1.8 * added flag for redisearch GC * moved env var setting to memcheck script * Update memcheck.sh * fixed misplaced env var setting * fixed bad command format * flag in circle ci instead of makefile Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * docker file for centos (#1104) * version bump Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Sceat <11330271+Sceat@users.noreply.github.com> Co-authored-by: Itamar Haber <itamar@redislabs.com> Co-authored-by: Ariel Shtul <ariel.shtul@redislabs.com> Co-authored-by: Timothy Rule <34501912+trulede@users.noreply.github.com> Co-authored-by: Rafi Einstein <rafi@redislabs.com> Co-authored-by: Rafi Einstein <raffapen@outlook.com> Co-authored-by: Christoph Zimmermann <40485189+chrisAtRedis@users.noreply.github.com>

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments

* Update op_node_by_id_seek.c Fix, wrong variable assignment. * Update value.c removed.boolean switch * Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) * Fixed compiler typo (#1097) Paragraph on OSX build should read "CXX" instead of "CPP". * Union bugfixes (#1052) * Improve scoping rules in validating UNION queries * Bugfix in uniquing column containing both nodes and edges * Add flow tests * PR fixes * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments * throw runtime-error on missing query parameters (#1100) * removed query parameters annotations (#1101) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) * changed RediSearch version to 1.8 (#1103) * changed RediSearch version to 1.8 * added flag for redisearch GC * moved env var setting to memcheck script * Update memcheck.sh * fixed misplaced env var setting * fixed bad command format * flag in circle ci instead of makefile Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * docker file for centos (#1104) * Skip NULL-valued properties in bulk loader (#1108) * Skip NULL-valued properties in bulk loader * Address PR comment, add missing logic in edge procesing * RED-4187: update RS version to 5.6.0 and run on Centos os on CI process (#1112) * Perform one-time transpose of traversed matrices (#1111) * Autoformat * WIP * No-op transpose nodes, working with memory leaks * WIP freeing and simplifying * WIP * Fix false positives on leak checking * Autoformat unit test file * Replace transpose op nodes with operands * Prune unused eval conditions, improve function logic * Fix unit tests * Partially address PR comments * PR fixes continued * PR fixes * Add handling for transpose operations in ApplyTranspose * Fix disconnect sequence in ApplyTranspose on operations Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Omp thread configuration (#1118) * Expose OMP_THREAD_COUNT as a module-level configuration * Update documentation * Improve comments * Refactor logic for module-level configuration * Fix PR comments * Enforce that module args are key-value pairs * PR fixes * PR fixes * redisearch 1.8.1 (#1121) * add doc action * remove internal folders * Add configurations to menu * Update deploy-docs.yaml (#1126) * Update deploy-docs.yaml (#1131) * Null PR for triggering docs (#1132) * Remove deploy-docs (moved to action) (#1140) * update min_redis_pack_version to 5.4.14 (#1142) * Transposed relations (#877) * maintain transposed matrices * Updated unit-tests * Post-rebase fixes * Revert FetchOperands changes * WIP * Unit test fixes * Simplify transpose matrix assignment * Add configuration param * Access config global to check for transposed matrices * Update documentation * Update graph logic to handle the absence of transposed matrices * Fix unit tests * Exit with error on unhandled parameter * Add flow test * Simplify fetch operands logic * Add abstraction layer to check for transposed relations * Partially address PR comments * Update FetchOperands logic * Address PR comments Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update index.md (#1143) * Exec plan cache (#1117) * added single threaded LRU cache * changed LRU logic. WIP after design review * fixed PR comments * removed DS_Store * changed pr comments for test priority queue * fixed priority_queue.h and linked_list.h comments. wip * fixed linked_list.c comments. wip * wip * Revert change to queue item sizing * WIP * wip * Remove linear insertion flag, simplify linked list * Refactor cache data structure implementations * Start adding logic for populating cache * added cache size config param * added cache API for graph context * did some clean ups * clone logic * done refactoring. unit tests pass * unit tests pass * wip * wip * test suit pass * test suit pass * fixed some memory leaks * wip * added cache tests * review ready * fixed PR comments * fixed PR comments * fixed PR comments * after rebase * fixed PR comments * fixing memory leaks * trying to avoid race Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * memceck race handling (#1152) * removed time.sleep from tests teardown * added async delete config * fixed config and makefile * moved memcheck to compiler flag * Avoid flushing matrices by maintaining separate transpose edge arrays (#1148) * Avoid flushing matrices by maintaining separate edge arrays in transposes * Properly update transpose matrices on single-edge deletion Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * version bump * fixed parameterized index scan (#1157) * added params support for indexed array lookup (#1159) * removed RedisModule_ReplyWithError from ast validations (#1160) * removed RedisModule_ReplyWithError from ast validations * removed char** reason from functions * refactored query_ctx for varargs Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added params support to id seek (#1164) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments * validate graph schema is encoded (#1168) * validate graph schema is encoded * fixed PR comment Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Resolve race in accessing/updating attribute maps (#1165) * Protect critical region of reading/updating GraphContext attributes * Improve lock coverage * Don't heap-alloc attribute ID values * Change to QueryCtx lock check * Bulk insert deadlock fix * Remove FindOrAddAttribute calls from critical region * Revert changes to QueryCtx * Revert changes to Graph * Introduce new rwlock for attribute mapping * Only lookup property ID once * Fix unit tests * Fix possible duplicate entry * Always retrieve attribute value in critical region * Address PR comments * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Move index iterator construction to first Consume call (#1169) * Move index iterator construction to first Consume call * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Christoph Zimmermann <40485189+chrisAtRedis@users.noreply.github.com> Co-authored-by: Omri Ben-Gidon <47712555+omrib1@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Itamar Haber <itamar@redislabs.com> Co-authored-by: swilly22 <roi@redislabs.com>

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6)

Bidirectional expand into (#1047)

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6)

* Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) (#1210) (cherry picked from commit 386a8e6) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update value.c (#1209) removed.boolean switch (cherry picked from commit e5ab233) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) (#1207) (cherry picked from commit cce2ebb) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) (#1206) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 0cf50df) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) (#1205) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 135e0bc) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * ReduceScan and ReduceTraversal respect variable scopes (#1070) (#1204) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 29b75d8) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Better compile-time checking for undefined variables (#1063) (#1203) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures (cherry picked from commit e62b823) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add cloud link and NOT conditions unary validation (#1214) * Guarantee that NOT conditions are unary (#1092) (#1208) (cherry picked from commit b56f98a) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * link to redis cloud Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6) * Skip NULL-valued properties in bulk loader (#1108) * Skip NULL-valued properties in bulk loader * Address PR comment, add missing logic in edge procesing (cherry picked from commit b62e802) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) (cherry picked from commit d6d6f8b) * added params support to id seek (#1164) (#1218) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments (cherry picked from commit bc609c3) * Move index iterator construction to first Consume call (#1169) (#1219) * Move index iterator construction to first Consume call * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 7853153) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com>

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6)

* Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) (#1210) (cherry picked from commit 386a8e6) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update value.c (#1209) removed.boolean switch (cherry picked from commit e5ab233) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) (#1207) (cherry picked from commit cce2ebb) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) (#1206) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 0cf50df) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) (#1205) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 135e0bc) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * ReduceScan and ReduceTraversal respect variable scopes (#1070) (#1204) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 29b75d8) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Better compile-time checking for undefined variables (#1063) (#1203) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures (cherry picked from commit e62b823) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add cloud link and NOT conditions unary validation (#1214) * Guarantee that NOT conditions are unary (#1092) (#1208) (cherry picked from commit b56f98a) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * link to redis cloud Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6) * Skip NULL-valued properties in bulk loader (#1108) * Skip NULL-valued properties in bulk loader * Address PR comment, add missing logic in edge procesing (cherry picked from commit b62e802) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) (cherry picked from commit d6d6f8b) * added params support to id seek (#1164) (#1218) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments (cherry picked from commit bc609c3) * Move index iterator construction to first Consume call (#1169) (#1219) * Move index iterator construction to first Consume call * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 7853153) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com>

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6)

* Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) (#1210) (cherry picked from commit 386a8e6) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update value.c (#1209) removed.boolean switch (cherry picked from commit e5ab233) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) (#1207) (cherry picked from commit cce2ebb) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) (#1206) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 0cf50df) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) (#1205) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 135e0bc) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * ReduceScan and ReduceTraversal respect variable scopes (#1070) (#1204) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 29b75d8) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Better compile-time checking for undefined variables (#1063) (#1203) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures (cherry picked from commit e62b823) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add cloud link and NOT conditions unary validation (#1214) * Guarantee that NOT conditions are unary (#1092) (#1208) (cherry picked from commit b56f98a) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * link to redis cloud Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6) * Skip NULL-valued properties in bulk loader (#1108) * Skip NULL-valued properties in bulk loader * Address PR comment, add missing logic in edge procesing (cherry picked from commit b62e802) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) (cherry picked from commit d6d6f8b) * added params support to id seek (#1164) (#1218) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments (cherry picked from commit bc609c3) * Move index iterator construction to first Consume call (#1169) (#1219) * Move index iterator construction to first Consume call * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 7853153) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com>

* Resolve race in accessing/updating attribute maps (#1165) (#1223) * Protect critical region of reading/updating GraphContext attributes * Improve lock coverage * Don't heap-alloc attribute ID values * Change to QueryCtx lock check * Bulk insert deadlock fix * Remove FindOrAddAttribute calls from critical region * Revert changes to QueryCtx * Revert changes to Graph * Introduce new rwlock for attribute mapping * Only lookup property ID once * Fix unit tests * Fix possible duplicate entry * Always retrieve attribute value in critical region * Address PR comments * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit d66affb) * Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) (#1210) (cherry picked from commit 386a8e6) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update value.c (#1209) removed.boolean switch (cherry picked from commit e5ab233) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) (#1207) (cherry picked from commit cce2ebb) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) (#1206) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 0cf50df) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) (#1205) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 135e0bc) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * ReduceScan and ReduceTraversal respect variable scopes (#1070) (#1204) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 29b75d8) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Better compile-time checking for undefined variables (#1063) (#1203) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures (cherry picked from commit e62b823) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add cloud link and NOT conditions unary validation (#1214) * Guarantee that NOT conditions are unary (#1092) (#1208) (cherry picked from commit b56f98a) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * link to redis cloud Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) (cherry picked from commit d6d6f8b) * added params support to id seek (#1164) (#1218) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments (cherry picked from commit bc609c3) * Skip NULL-valued properties in bulk loader (#1108) (#1216) * Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) (#1210) (cherry picked from commit 386a8e6) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update value.c (#1209) removed.boolean switch (cherry picked from commit e5ab233) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) (#1207) (cherry picked from commit cce2ebb) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) (#1206) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 0cf50df) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) (#1205) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 135e0bc) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * ReduceScan and ReduceTraversal respect variable scopes (#1070) (#1204) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 29b75d8) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Better compile-time checking for undefined variables (#1063) (#1203) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures (cherry picked from commit e62b823) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add cloud link and NOT conditions unary validation (#1214) * Guarantee that NOT conditions are unary (#1092) (#1208) (cherry picked from commit b56f98a) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * link to redis cloud Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments (cherry picked from commit d46c6b6) * Skip NULL-valued properties in bulk loader (#1108) * Skip NULL-valued properties in bulk loader * Address PR comment, add missing logic in edge procesing (cherry picked from commit b62e802) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) (cherry picked from commit d6d6f8b) * added params support to id seek (#1164) (#1218) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments (cherry picked from commit bc609c3) * Move index iterator construction to first Consume call (#1169) (#1219) * Move index iterator construction to first Consume call * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit 7853153) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> * add GTM (#1237) (cherry picked from commit 8cc66d8) * fixed graph context creation * Update index on change (#1225) * WIP * Fix compile errors * Only update indexes when necessary * refined update entity update eval * WIP * WIP * Compilation fixes, edge updates * Add freeing logic * Update comments * some minor changes to op_update * Address PR comments * Address PR comments * always get updated node label id Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: DvirDukhan <dvir@redislabs.com> (cherry picked from commit e712193) * Update index on change (#1225) * WIP * Fix compile errors * Only update indexes when necessary * refined update entity update eval * WIP * WIP * Compilation fixes, edge updates * Add freeing logic * Update comments * some minor changes to op_update * Address PR comments * Address PR comments * always get updated node label id Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: DvirDukhan <dvir@redislabs.com> (cherry picked from commit e712193) * memceck race handling (#1152) * removed time.sleep from tests teardown * added async delete config * fixed config and makefile * moved memcheck to compiler flag (cherry picked from commit b7d0a38) Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com> Co-authored-by: Guy Korland <gkorland@gmail.com>

* Update op_node_by_id_seek.c Fix, wrong variable assignment. * Update value.c removed.boolean switch * Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) * Fixed compiler typo (#1097) Paragraph on OSX build should read "CXX" instead of "CPP". * Union bugfixes (#1052) * Improve scoping rules in validating UNION queries * Bugfix in uniquing column containing both nodes and edges * Add flow tests * PR fixes * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments * throw runtime-error on missing query parameters (#1100) * removed query parameters annotations (#1101) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) * changed RediSearch version to 1.8 (#1103) * changed RediSearch version to 1.8 * added flag for redisearch GC * moved env var setting to memcheck script * Update memcheck.sh * fixed misplaced env var setting * fixed bad command format * flag in circle ci instead of makefile Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * docker file for centos (#1104) * Skip NULL-valued properties in bulk loader (#1108) * Skip NULL-valued properties in bulk loader * Address PR comment, add missing logic in edge procesing * RED-4187: update RS version to 5.6.0 and run on Centos os on CI process (#1112) * Perform one-time transpose of traversed matrices (#1111) * Autoformat * WIP * No-op transpose nodes, working with memory leaks * WIP freeing and simplifying * WIP * Fix false positives on leak checking * Autoformat unit test file * Replace transpose op nodes with operands * Prune unused eval conditions, improve function logic * Fix unit tests * Partially address PR comments * PR fixes continued * PR fixes * Add handling for transpose operations in ApplyTranspose * Fix disconnect sequence in ApplyTranspose on operations Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Omp thread configuration (#1118) * Expose OMP_THREAD_COUNT as a module-level configuration * Update documentation * Improve comments * Refactor logic for module-level configuration * Fix PR comments * Enforce that module args are key-value pairs * PR fixes * PR fixes * redisearch 1.8.1 (#1121) * add doc action * remove internal folders * Add configurations to menu * Update deploy-docs.yaml (#1126) * Update deploy-docs.yaml (#1131) * Null PR for triggering docs (#1132) * Remove deploy-docs (moved to action) (#1140) * update min_redis_pack_version to 5.4.14 (#1142) * Transposed relations (#877) * maintain transposed matrices * Updated unit-tests * Post-rebase fixes * Revert FetchOperands changes * WIP * Unit test fixes * Simplify transpose matrix assignment * Add configuration param * Access config global to check for transposed matrices * Update documentation * Update graph logic to handle the absence of transposed matrices * Fix unit tests * Exit with error on unhandled parameter * Add flow test * Simplify fetch operands logic * Add abstraction layer to check for transposed relations * Partially address PR comments * Update FetchOperands logic * Address PR comments Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Update index.md (#1143) * Exec plan cache (#1117) * added single threaded LRU cache * changed LRU logic. WIP after design review * fixed PR comments * removed DS_Store * changed pr comments for test priority queue * fixed priority_queue.h and linked_list.h comments. wip * fixed linked_list.c comments. wip * wip * Revert change to queue item sizing * WIP * wip * Remove linear insertion flag, simplify linked list * Refactor cache data structure implementations * Start adding logic for populating cache * added cache size config param * added cache API for graph context * did some clean ups * clone logic * done refactoring. unit tests pass * unit tests pass * wip * wip * test suit pass * test suit pass * fixed some memory leaks * wip * added cache tests * review ready * fixed PR comments * fixed PR comments * fixed PR comments * after rebase * fixed PR comments * fixing memory leaks * trying to avoid race Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * memceck race handling (#1152) * removed time.sleep from tests teardown * added async delete config * fixed config and makefile * moved memcheck to compiler flag * Avoid flushing matrices by maintaining separate transpose edge arrays (#1148) * Avoid flushing matrices by maintaining separate edge arrays in transposes * Properly update transpose matrices on single-edge deletion Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * fixed parameterized index scan (#1157) * added params support for indexed array lookup (#1159) * removed RedisModule_ReplyWithError from ast validations (#1160) * removed RedisModule_ReplyWithError from ast validations * removed char** reason from functions * refactored query_ctx for varargs Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added params support to id seek (#1164) * added params support to id seek * reduce to scalar with runtime * fixed PR comments * fixed PR comments * validate graph schema is encoded (#1168) * validate graph schema is encoded * fixed PR comment Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Resolve race in accessing/updating attribute maps (#1165) * Protect critical region of reading/updating GraphContext attributes * Improve lock coverage * Don't heap-alloc attribute ID values * Change to QueryCtx lock check * Bulk insert deadlock fix * Remove FindOrAddAttribute calls from critical region * Revert changes to QueryCtx * Revert changes to Graph * Introduce new rwlock for attribute mapping * Only lookup property ID once * Fix unit tests * Fix possible duplicate entry * Always retrieve attribute value in critical region * Address PR comments * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Move index iterator construction to first Consume call (#1169) * Move index iterator construction to first Consume call * Address PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * fix graph creation example (#1171) * fix graph creation example * fixed errors in example * single flow (#1177) * added explain and profile options as invalid options in redisgraph (#1184) * added explain and profile options as invalid options in redisgraph query string * fixed PR comments * moved tests * Update Dockerfile (#1188) * Adding redis cloud pro to quick start Please hold back merging this PR. Thanks * fix typo in anker * enable search GC (#1194) * aggregated slowlog should maintain original timestamps (#1199) * correct link to redis cloud * Aumation auth moved to be token based (#1192) * Aumation auth moved to be token based * Update config.yml * Update config.yml * relay on search replace add functionality to delete existing documents (#1226) * line length 80 (#1227) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Emit compile time error on creation of undirected edges (#1212) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * redisearch 1.8.2 (#1229) * redisearch 1.8.2 * linked search cleanup function * add GTM (#1239) * Update index on change (#1225) * WIP * Fix compile errors * Only update indexes when necessary * refined update entity update eval * WIP * WIP * Compilation fixes, edge updates * Add freeing logic * Update comments * some minor changes to op_update * Address PR comments * Address PR comments * always get updated node label id Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: DvirDukhan <dvir@redislabs.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Christoph Zimmermann <40485189+chrisAtRedis@users.noreply.github.com> Co-authored-by: Omri Ben-Gidon <47712555+omrib1@users.noreply.github.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Itamar Haber <itamar@redislabs.com> Co-authored-by: Martin Rauscher <hades32@gmail.com> Co-authored-by: Pieter Cailliau <pieter.cailliau@gmail.com>

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments

jeffreylovitz added the bug label Apr 1, 2020

jeffreylovitz added this to the RedisGraph 2.2 milestone Apr 1, 2020

jeffreylovitz requested review from swilly22 and DvirDukhan April 1, 2020 20:18

jeffreylovitz self-assigned this Apr 1, 2020

DvirDukhan reviewed Apr 1, 2020

View reviewed changes

DvirDukhan reviewed Apr 2, 2020

View reviewed changes

jeffreylovitz force-pushed the bidirectional-expand-into branch from 216f1e7 to 9828773 Compare April 9, 2020 17:53

jeffreylovitz force-pushed the bidirectional-expand-into branch from 9828773 to 3459fd9 Compare April 22, 2020 20:13

jeffreylovitz mentioned this pull request May 12, 2020

removed redundant iterator #1095

Closed

swilly22 requested changes May 13, 2020

View reviewed changes

jeffreylovitz added 7 commits May 13, 2020 10:54

wip

9f235e8

Remove unnecessary iterator

83c4f9a

WIP

07e695f

Fix bidirectional ExpandInto, shared edge populating logic

75b8d74

PR fixes

400c9b4

Post-rebase fixes

46d9901

Fix PR comments

8aaf931

jeffreylovitz force-pushed the bidirectional-expand-into branch from 3459fd9 to 8aaf931 Compare May 13, 2020 15:27

swilly22 approved these changes May 13, 2020

View reviewed changes

swilly22 merged commit d46c6b6 into master May 13, 2020

swilly22 deleted the bidirectional-expand-into branch May 13, 2020 18:06

DvirDukhan pushed a commit that referenced this pull request May 21, 2020

Bidirectional expand into (#1047)

98bf9bb

* wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments

DvirDukhan added a commit that referenced this pull request Jul 9, 2020

Merge pull request #1215 from RedisGraph/merge_1047_into_2.0

e952277

Bidirectional expand into (#1047)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bidirectional expand into #1047

Bidirectional expand into #1047

jeffreylovitz commented Apr 1, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 1, 2020

jeffreylovitz Apr 2, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 1, 2020

jeffreylovitz Apr 2, 2020

DvirDukhan Apr 1, 2020

DvirDukhan Apr 2, 2020

jeffreylovitz Apr 2, 2020

swilly22 left a comment

swilly22 May 13, 2020

swilly22 May 13, 2020

swilly22 May 13, 2020

swilly22 May 13, 2020

swilly22 May 13, 2020

swilly22 May 13, 2020

swilly22 May 13, 2020

Bidirectional expand into #1047

Bidirectional expand into #1047

Conversation

jeffreylovitz commented Apr 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swilly22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment