Optional match #1043

jeffreylovitz · 2020-03-30T20:51:59Z

No description provided.

swilly22

First review, I've looked at doc and tests only.

swilly22 · 2020-03-31T08:01:11Z

tests/flow/test_optional_match.py

+        # The OPTIONAL MATCH exactly repeats the MATCH, producing identical results.
+        query_without_optional = """MATCH (a)-[e]->(b) RETURN a.v, b.v, TYPE(e) ORDER BY a.v, b.v"""
+        result_without_optional = redis_graph.query(query_without_optional)
+        self.env.assertEquals(actual_result.result_set, result_without_optional.result_set)


Isn't this covered by:

expected_result = [['v1', 'v2', 'E1'], ['v2', 'v3', 'E2']] self.env.assertEquals(actual_result.result_set, expected_result)

Yes, but I liked having the opportunity to test an OPTIONAL MATCH result against a MATCH result. Do you want this removed?

no let's keep it.

tests/flow/test_null_handling.py

swilly22 · 2020-03-31T08:14:07Z

tests/flow/test_null_handling.py

+        self.env.assertEquals(actual_result.result_set, expected_result)
+
+    # Path functions should handle null inputs appropriately.
+    def test05_null_named_path_function_inputs(self):


Should we test for every single function we have?
What about aggregation function and procedures?
can we at-least test for some representatives from each?

We have 60 non-aggregate functions, and very few have unique null handling - I don't really want to write that many tests. We should have some aggregate and procedure tests, I will add those!

tests/flow/test_null_handling.py

swilly22 · 2020-03-31T11:08:26Z

docs/commands.md

+RETURN p, w, c"
+```
+
+All `Person` nodes are returned, as well as any `WORKS_AT` relations and `Company` nodes that can be resolved and satisfy the `start_date` constraint. For each `Person` that does not resolve the optional pattern, the person will be returned as normal and the non-matching elements will be returned as null.


So if there are multiple connection between a person p and a number of companies say 10
and out of those 10 only 4 satisfy the criteria how many null will be returned?

0 nulls - if p has at least one valid connection, no nulls will be produced. If there is another person p with no valid connections, it will return that p once with null w and c.

swilly22 · 2020-03-31T11:09:09Z

docs/commands.md

+
+In this case, `w.department` and `ID` will be returned if the OPTIONAL MATCH was successful, and will be null otherwise.
+
+Clauses like SET, CREATE, and DELETE will ignore null inputs and perform the expected updates on real inputs. One exception to this is that attempting to create a relation with a null endpoint will cause an error:


What about MERGE ?

MERGE belongs here as well, I'll add it. Internally, this is all handled by the MergeCreate op, the logic is the same as Create.

tests/flow/test_optional_match.py

DvirDukhan · 2020-03-30T21:42:40Z

tests/flow/test_optional_match.py

+                           ['v1', 'v4', None]]
+        self.env.assertEquals(actual_result.result_set, expected_result)
+
+    # TODO ExpandInto doesn't evaluate bidirectionally properly


still relevant?

Yes, this is the unrelated issue #1036. I'll uncomment this test once that's resolved.

tests/flow/test_null_handling.py

src/arithmetic/entity_funcs/entity_funcs.c

src/arithmetic/path_funcs/path_funcs.c

DvirDukhan · 2020-03-31T11:27:59Z

src/execution_plan/record.c

+		// Null scalar values are expected here; otherwise fall through.
+		if(SIValue_IsNull(r->entries[idx].value.s)) return NULL;
+	default:
+		assert("encountered unexpected type in Record; expected Node" && false);


please add the type

Since this is a string literal, the only way to do that would be to have separate assertions for each case, which feels like overkill to me. (Even the current case statement layout kind of feels like overkill to me.) Thoughts?

DvirDukhan · 2020-03-31T11:28:08Z

src/execution_plan/record.c

+		// Null scalar values are expected here; otherwise fall through.
+		if(SIValue_IsNull(r->entries[idx].value.s)) return NULL;
+	default:
+		assert("encountered unexpected type in Record; expected Edge" && false);


please add the type

swilly22 · 2020-04-02T07:23:53Z

tests/flow/test_null_handling.py

+
+    # SET should update attributes on non-null entities and ignore null entities.
+    def test03_set_null(self):
+        query = """MATCH (a) OPTIONAL MATCH (a)-[nonexistent_edge]->(nonexistent_node) SET a.v2 = true, nonexistent_node.v2 = true RETURN a.v2, nonexistent_node.v2"""


Please add a.v3 = nonexistent_node.v3

swilly22 · 2020-04-02T07:25:55Z

tests/flow/test_null_handling.py

+        self.env.assertEquals(actual_result.result_set, expected_result)
+
+    # List functions should handle null inputs appropriately.
+    def test07_null_list_function_inputs(self):


this test seems to be out of place, I expect it to show up in a list test file.

swilly22 · 2020-04-02T07:37:33Z

src/execution_plan/record.c

@@ -89,18 +89,36 @@ RecordEntryType Record_GetType(const Record r, int idx) {
 }

 SIValue Record_GetScalar(Record r, int idx) {
-	r->entries[idx].type = REC_TYPE_SCALAR;
+	assert(r->entries[idx].type == REC_TYPE_SCALAR);


Remove Record_GetScalar replace calls with Record_Get

swilly22 · 2020-04-02T07:41:41Z

src/execution_plan/record.c

+	case REC_TYPE_UNKNOWN:
+		return SI_NullVal();


assert(false && "Invalid entry type");

swilly22 · 2020-04-02T10:47:11Z

src/execution_plan/ops/op_node_by_label_scan.c

 	// Populate the Record with the graph entity data.
-	Graph_GetNode(op->g, node_id, n);
+	Node n = {};


Node n = {0};

swilly22 · 2020-04-02T11:01:51Z

src/arithmetic/path_funcs/path_funcs.c

+	for(uint i = 1; i < argc; i++) {
+		// If any element of the path does not exist, the entire path is invalid and NULL should be returned.
+		if(SI_TYPE(argv[i]) == T_NULL) return SI_NullVal();
+	}
+


To avoid this extra scan,
perform this logic within the following loop,
when finding a NULL entity, free path and return NULL

swilly22 · 2020-04-02T11:15:32Z

src/arithmetic/arithmetic_expression.c

@@ -344,8 +344,12 @@ static bool _AR_EXP_UpdateEntityIdx(AR_OperandNode *node, const Record r) {

 static AR_EXP_Result _AR_EXP_EvaluateProperty(AR_ExpNode *node, const Record r, SIValue *result) {
 	RecordEntryType t = Record_GetType(r, node->operand.variadic.entity_alias_idx);
-	// Property requested on a scalar value.
-	if(!(t & (REC_TYPE_NODE | REC_TYPE_EDGE))) {
+	if(t == REC_TYPE_UNKNOWN) {


if(!(t & (REC_TYPE_NODE | REC_TYPE_EDGE))) { if(t == REC_TYPE_UNKNOWN) { /* If we attempt to access a null value as a graph entity (due to a * scenario like a failed OPTIONAL MATCH), return a null value. */ *result = SI_NullVal(); return EVAL_OK; } else { /* Attempted to access a scalar value as a map. * Set an error and invoke the exception handler. */ char *error; SIValue v = Record_GetScalar(r, node->operand.variadic.entity_alias_idx); asprintf(&error, "Type mismatch: expected a map but was %s", SIType_ToString(SI_TYPE(v))); QueryCtx_SetError(error); // Set the query-level error. return EVAL_ERR; } }

swilly22 · 2020-04-02T11:32:32Z

src/ast/ast_validations.c

+		if(type == CYPHER_AST_MATCH) {
+			// Check whether this match is optional.
+			bool current_clause_is_optional = cypher_ast_match_is_optional(clause);
+			// If it is not and we have already processed an optional match, emit an error.


"// If it is not and we..." ?

swilly22 · 2020-04-02T11:37:50Z

src/execution_plan/ops/op_optional.h

+/* If Optional's child produces records, Optional emits them without modification.
+ * If the child produces no records, Optional emits an empty Record exactly once. */


Please give a short introduction explaining this operation

swilly22 · 2020-04-02T18:19:43Z

src/execution_plan/ops/op_optional.c

+static void OptionalFree(OpBase *ctx) {
+	Optional *op = (Optional *)ctx;
+	op->emitted_record = false;
+}


No need, please remove.

swilly22 · 2020-04-03T10:54:05Z

src/execution_plan/ops/op_apply.c

+	Apply *op = (Apply *)opBase;
+	/* The op's bound branch and optional match branch have already been built as
+	 * the Apply op's first and second child ops, respectively. */
+	op->bound_branch = opBase->children[0];
+	op->rhs_branch = opBase->children[1];


My only concern is optimisations which might shuffle op's children array.

I can't think of any scenario that will violate this assumption currently. For now the direct child of the RHS branch is guaranteed to be an Optional op, so we could explicitly check for that, but I'd prefer to keep Apply's logic more generic so that it can be used for other purposes in the future.

swilly22 · 2020-04-03T10:57:52Z

src/execution_plan/ops/op_apply.c

-	}
+	// Locate branch's Argument op tap.
+	op->op_arg = (Argument *)ExecutionPlan_LocateOp(op->rhs_branch, OPType_ARGUMENT);
+	assert(op->op_arg && op->op_arg->op.childCount == 0);


op->op_arg->op.childCount == 0 this is not the place to make sure OP_ARG has no children.

swilly22 · 2020-04-03T11:04:07Z

src/execution_plan/ops/op_apply.c

-	// Clone the left-hand record
-	Record r = OpBase_CloneRecord(op->lhs_record);
+			// Successfully pulled a new Record, propagate to the top of the RHS branch.
+			if(op->op_arg) Argument_AddRecord(op->op_arg, OpBase_CloneRecord(op->r));


ApplyInit verified that op->op_arg isn't NULL
assert(op->op_arg && op->op_arg->op.childCount == 0);

swilly22 · 2020-04-03T11:17:10Z

src/execution_plan/ops/op_expand_into.c

@@ -58,6 +58,7 @@ static void _populate_filter_matrix(OpExpandInto *op) {
 		/* Update filter matrix F, set row i at position srcId
 		 * F[i, srcId] = true. */
 		Node *n = Record_GetNode(r, op->srcNodeIdx);
+		assert(n && "failed to resolve source node for ExpandInto");


You're guarantee n will exists, see if(!Record_GetNode(childRecord, op->srcNodeIdx)) { consume function.

swilly22 · 2020-04-03T11:21:28Z

src/execution_plan/ops/op_expand_into.c

+		if(!srcNode || !destNode) {
+			// An endpoint may not resolve if an Optional op failed to match,
+			// in which case we can skip this record.
+			// Mark as NULL to avoid double free.
+			op->records[op->recordCount] = NULL;
+			continue;
+
+		}


srcNode is guarantee to exists, see consume function

it is wasteful to see if destNode exists in the record at this point in time, it would be much better to check for destNode within the consume function. we would like to consider records where both src and dest exists.

swilly22 · 2020-04-03T11:22:19Z

src/execution_plan/ops/op_expand_into.c

+			if(!Record_GetNode(childRecord, op->srcNodeIdx)) {
+				/* The child Record may not contain the source node in scenarios like
+				 * a failed OPTIONAL MATCH. In this case, delete the Record and try again. */
+				OpBase_DeleteRecord(childRecord);
+				op->recordCount--;
+				continue;
+			}


Same check should be performed on destNode

swilly22 · 2020-04-03T11:26:14Z

src/execution_plan/ops/op_optional.h

+#include "op.h"
+#include "../execution_plan.h"
+
+/* Optional is an operation that manages the output of its child op tree rather


Perfect!
we should have such an introduction for every operation!

swilly22 · 2020-04-03T11:28:56Z

src/execution_plan/ops/op_update.c

 	for(uint i = 0; i < op->pending_updates_count; i++) {
 		EntityUpdateCtx *ctx = &op->pending_updates[i];
 		// Map the attribute key if it has not been encountered before
 		if(ctx->attr_id == ATTRIBUTE_NOTFOUND) {
 			ctx->attr_id = GraphContext_FindOrAddAttribute(op->gc, ctx->attribute);
 		}
 		if(ctx->entity_type == GETYPE_NODE) {
-			_UpdateNode(op, ctx);
+			properties_set += _UpdateNode(op, ctx);


_UpdateNode returns bool, I find it strange to be adding bool to uint
either update _UpdateNode and _UpdateEdge return type or have an if condition here

👍 Switched return types to ints.

swilly22 · 2020-04-03T11:32:16Z

src/execution_plan/ops/op_update.c

 			RecordEntryType t = Record_GetType(r, update_expression->record_idx);
+			// If the expected entity was not found, make no updates but do not error.
+			if(t == REC_TYPE_UNKNOWN) continue;
+			// Make sure we're updating either a node or an edge.
 			assert(t == REC_TYPE_NODE || t == REC_TYPE_EDGE);


I feel like this is going to be an open issue at some point, consider emitting a run-time error.

swilly22 · 2020-04-03T11:35:09Z

src/execution_plan/record.c

@@ -125,7 +140,7 @@ GraphEntity *Record_GetGraphEntity(const Record r, int idx) {
 	case REC_TYPE_EDGE:
 		return (GraphEntity *)Record_GetEdge(r, idx);
 	case REC_TYPE_SCALAR:
-		return (GraphEntity *)(Record_GetScalar(r, idx).ptrval);
+		return (GraphEntity *)(Record_Get(r, idx).ptrval);


What if Record_Get(r, idx) returns a boolean SIValue ?

swilly22

Thrilled to approve this PR!

* Support float inputs for modulo computations (#895) * Add references * Update mkdocs.yml * Update References.md * fixed ast mapping for path filter (#896) * fixed ast mapping for path filter * renamed tests. open a new redis graph client per tests Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added mutex per matrix (#898) * added mutex per matrix * fixed PR comments * updated license headers to 2020 (#902) * Preserve order of op's children array when introducing index scans (#912) * Fix memory leaks (#917) * Fix memory leaks * Use original logic for MarkWriter * added filter tree clone (#915) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Optimize cp (#906) * added optimize cp * fixed a bug. added test * add optimization_util files * fixed PR comments * fixed PR comments * added multiple branch cp optimization * fixed PR comment * added in place replacement at ExecutionPlan_RemoveOp * added test for cp optimization and semi apply * fixed PR comments * Added non existing entity runtime error (#919) * Added non existing entity runtime error * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * RediSearch query error reporting (#925) * added filter tree compaction (#922) * added filter tree compaction * fixed PR comments * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Handle both possible execution orders in concurrent rename test (#926) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Refferencing a RedisGraph javascript client library (#928) * Improve client libraries sections on both readme and clients.md Clients libraries are subject to repetitive edition, it seems better to have a concise formatting * refferencing my own contribution to clients libraries Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Rust client (#931) * Update README.md * Update clients.md * intoduced fpClone in OpBase (#930) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Apply deplete match stream (#913) * Deplete Apply op's match stream for every left-hand Record * Disallow ExpandInto ops on variables beneath Apply ops * Add tests * PR fixes * Argument op holds one Record * PR fixes * Improve logic for building OpArgument modifies arrays * standardize logic for building tmp ExecutionPlans * Remove sub-ExecutionPlan logic * Fix CondTraverseReset routine * Fix ExpandIntoReset * Fix variable-length QueryGraph pointer * Always reset match branch * Add explanatory comment * PR fixes * PR fixes * raxValues returns void pointer array Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * increased allowed parameters count to UNIT_MAX. fixed a bug in query_ctx (#933) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * introduce mapping between record entries and projected columns (#936) * added arr clone with cb (#937) * added arr clone with cb * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Leak fixes (#935) * Close Redis key handles * Fix memory leak in checking whether procedures are read-only * Improve explanatory comment Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * label matrix should be fetch right before eval (#938) * More sensible function signature for SIValue_Free (#941) * Resolve memory leaks on Path SIValues (#940) * Resolve memory leaks on Path SIValues * Update op_delete.c * Update op_cond_var_len_traverse.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * GraphBLAS 3.2.0 (#942) * GraphBLAS 3.2.0 * updated makefiles * Remove unnecessary GraphBLAS build flag Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Simplify OpAggregate logic, remove unnecessary struct members (#947) * Simplify OpAggregate logic, remove unnecessary struct members * Update op_aggregate.c * Update op_aggregate.c * Update op_aggregate.c * Update op_aggregate.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add compile-time error for unsupported AST node types (#944) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * removed unused operations and types (#950) * remove GraphBLAS cacheing from CI (#953) * switch from OR AND semiring to ANY PAIR (#955) * using the structured semiring we can combine relation and relation ma… (#964) * using the structured semiring we can combine relation and relation mapping into a single matrix * address PR comments * Update graph.c * decoupled result set from execution plan (#929) * decoupled result set from execution plan * after rebase merge * fixed PR comments * Restored resulte set * fixed PR comments * fixed PR comments * fixed PR comments * Update ast.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Always free emptied space on AlgebraicExpression replacement (#986) * Fix leak on full-text index queries with syntax error (#985) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix leak on index scan IN optimization (#984) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Minor updates to bulk deletion (#994) * Remove redundant clone from bulk insertion of string props (#993) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix memory leaks on RDB-loaded strings (#991) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add flow test to validate (#967) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Update outdated logic for aggregate groups (#968) * Update outdated logic for aggregate groups * Update group.c * Update op_aggregate.c * Update op_aggregate.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added all_node_scan clone (#960) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added op argument clone (#959) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added cartesian product clone (#963) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added expand into clone (#973) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added project op clone (#980) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added skip op clone (#983) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added op unwind clone (#988) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added limit op clone (#976) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added proc call clone (#979) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added filter op clone (#974) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added delete op clone (#970) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added distinct op clone (#972) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * add hash join op clone (#989) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added semi apply clone (#982) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added results op clone (#981) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added join op clone (#975) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added apply multiplexer clone (#961) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added cond var len traverse clone (#966) * added cond var len traverse clone * added conditional traverse clone * fixed PR comments * Reintroduce logic for freeing memory on run-time errors (#992) * Reintroduce logic for freeing memory on run-time errors * Remove VolatileRecord logic for freeing after run-time errors * Fix memory leaks on run-time errors in OpProject Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Free internal edge arrays on relationship matrix deletion (#997) * Free internal edge arrays on relationship matrix deletion * Update graph.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add debug function to print query (#995) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added clone for create, merge, merge_create, update (#969) * added clone for create, merge, merge_create, update * Update ast_shared.c * Update ast_shared.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Agg func fixes (#946) * Fix memory leaks in Collect function * Fix leak in aggregate func's SIValue result * Fix leak in children of aggregate function call * Simplify variable-length path free logic * Improve ownership logic in Collect, add explanatory comments * PR fix Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added label scan clone (#978) * added label scan clone * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added id seek clone (#977) * added id seek clone * added better comment on the range clone * Update op_node_by_id_seek.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added op aggreage clone (#958) * added op aggreage clone * fixed PR comments * Update op_aggregate.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add install instructions for OpenMP (#1006) * Add install instructions for OpenMP * Update README * added op sort clone (#987) * added op sort clone * fixed PR comments * removed free_list logic * Update execution_plan.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * re-enable graph.profile (#1004) * Re-enable GraphBLAS circle-ci cache (#1007) * Fix leak on projected heap-allocated graph entities (#996) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * GRAPH.SLOWLOG (#897) * slowlog WIP * slowlog per graph * updated docs * Add longer-running query to slowlog flow test * avoid race, log only GRAPH.QUERY Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * Use NOP label scan if range iterator construction fails (#1001) * Use NOP label scan if range iterator construction fails * deplete iterator for invalid range * dont access op consume function directly Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * removed redisearch optimization (#1010) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Note ReplicaOf memory constraint in Rampfile (#1011) * Automate leak check (#1002) * Add make memcheck rule * Add Circle memcheck job * Don't use Docker image for automated leak checking * Enable log names by migrating flow test Env initialization * Disable invalid TCK scenarios of issue #945 * Run memory test without optimizations after packaging artifacts * Add suppressions for leaks in flush-vs-shutdown race Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Write longer-running query for slowlog test (#1012) * OPType bitmask switched to contiguous enum (#1013) * OPType bitmask switched to contiguous enum * PR fixes * Clean up op tree modification code * Add static array of all scan ops * Remove unnecessary conditional * Remove static type array for unoptimized scans * Validate arity per-command rather than globally (#1023) * Ar exp param (#990) * wip * wip * after rebase * updated libcypher parser * added AR_EXP_PARAM * fixed PR comments * fixed PR comments * fixed PR comments * fixed PR comments * Update arithmetic_expression.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix invalid op sequence when introducing index scans (#1028) * Fix invalid op sequence when partially replacing filters with index scans * Simplify op-freeing logic in utilizeIndices * Automate testing against enterprise v5.4.14 * push down transpose operations (#1032) * push down transpose operations * added test for transposed bi-directional edge * Update algebraic_expression_optimization.c * allow skip and limit to be parametes. (#1020) * wip * test pass * added skip limit params test * added failure test * fixed PR comments * fixed online review comments * restored traverse record cap * fixing memory leaks * fixed PR comments * changed redisearch version to 1.6.11 (#1035) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added datablock out of order ops (#1022) * added datablock out of order ops * fixed PR comments * Update oo_datablock.h * Update oo_datablock.c * fixed PR comments * fixed PR comments Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Adds versioned documentation (#1034) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * deploy on package release only (#1049) * deploy on package release only * fixed yml * changed workflows * changed workflows * fixed review comments * Optional match (#1043) * Enable TCK tests * Introduce Optional and Apply ops * Modify mock AST logic * Emit error on queries beginning with OPTIONAL MATCH * Return null on property accesses of null graph entity * Disallow OPTIONAL MATCH...MATCH queries * Fix OPTIONAL filter placement * Enable TCK tests * NULL handling for path functions * NULL handling for GraphEntity and list functions * WIP improve mock AST logic * Add flow tests * Improve AST mock logic * Error handling for SET and CREATE on null entities * Record_Get refactor * Test null handling * Minor cleanup * Add documentation * Simplify toPath null handling * Improve comments * Allow OPTIONAL MATCH as first clause * Simplify null-checking logic in create ops * Use branch of Python client for testing * PR fixes * PR fixes * Remove Record_GetScalar interface * PR fixes * PR fixes * Add demo query for OPTIONAL MATCH * Use standard Python client for automation * Emit all columns as SIValues in compact formatter * Improve flow test for null entities in first result Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * added named path example (#1048) * added named path example * fixed pr comments * Better compile-time checking for undefined variables (#1063) * Improve AST validations to capture undefined variables * Propagate errors in nested AR_EXP_Evaluate failures * Add path comparison, streamline hashing logic (#1056) * Value comparison call correctly compares paths * Fix bug in DISTINCT paths and arrays * Fix memory leak in ValueHashJoin * Remove conditionals from Record hashing logic * Improve test coverage * Simplify memory management in ValueHashJoin Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Update README.md * Update index.md * Update README.md forum (#1073) * Update README.md forum * ReduceScan and ReduceTraversal respect variable scopes (#1070) * ReduceScan and ReduceTraversal respect variable scopes * Fix source lookup * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * make memcheck rule prints full log for files with leaks (#1072) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Error properly on graph accesses of non-graph keys (#1069) * Error properly on graph accesses of non-graph keys * Explicitly free QueryCtx on failed delete * Update cmd_delete.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * LabelScan with child reads data correctly after reset (#1086) * Guarantee that NOT conditions are unary (#1092) * is -> ID in docs (#1090) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Alpine variant of Dockerfile (#1087) * Calling autogen.sh fails if execute permissions are not set. Rather than do that, explicitly use "sh" to run it. * Explicitly include sys/types.h Several types (u_intN_t) are defined in this header, and on some systems not included indirectly. Therefore include the file directly. * Add an Alpine variant of the Dockerfile. * Change alpine image basis to Redis v6. Tag the image as alpine (without redis version). Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Redis 6 in builds (#1060) * changed dokerfile and ci * New installation scheme in Dockerfile * fixes 1 * Review fixes 1 Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Rafi Einstein <rafi@redislabs.com> * Fix for RAMP failure (#1093) * Graph encoding v7 - support replica-of (#1054) * added entities threshold to config * wip * added commit flow, without meta type * new graphmeta type * wip * added decode v6 * added new encoding flow * fixed edge encoding * done decoding * added entities threshold to config * wip * added commit flow, without meta type * wip * added decode v6 * added new encoding flow * fixed edge encoding * added tags * wip * wip * tests pass * wip * moved to redis 6, solved flushall. moved to uuid keys. adeded decode context * wip * moved to redis 6 events to handle keyspace * added graph pending replication * wip * wip * changed tagging * wip * tested for redis6 and redis5 * fixed ubuntu build error. added comments * changed get redis major version location * pr comments. wip * refactor. wip * fixed PR comments * fixed memeory leak * added delete guards * PR comments, wip * removed meta context type. removed uuid from meta key name * fixed edges array encoding. added tests * added skip test for redis 5 * Simplify meta key construction * fixed PR comments * added aux fields * added module replicating error. WIP * added v4, v6 rdb decode test * fixed v4 decode memory leak * fixed PR comments * fixe PR comments * fixed PR comments * fixed PR comments * fixed PR comments * added tear down to flow test to avoid race between RLTest and RG * Fix memory leak in v4 deserialization * huge refactor: single encoder/decoder logic. created config object. create redis-server version object. May support redis 5 * wip * changed virtual keys encoding logic. wip * tested on redis5. logs added * added uuid to meta keys * removed server version checking in flow tests * changed ramp * fixed redis server validation * fixed PR comments * fixed PR comments Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Update op_node_by_id_seek.c Fix, wrong variable assignment. * Update value.c removed.boolean switch * Add suppression for erroneous leak reports after DEBUG RELOAD (#1094) * Fixed compiler typo (#1097) Paragraph on OSX build should read "CXX" instead of "CPP". * Union bugfixes (#1052) * Improve scoping rules in validating UNION queries * Bugfix in uniquing column containing both nodes and edges * Add flow tests * PR fixes * PR fixes Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Bidirectional expand into (#1047) * wip * Remove unnecessary iterator * WIP * Fix bidirectional ExpandInto, shared edge populating logic * PR fixes * Post-rebase fixes * Fix PR comments * throw runtime-error on missing query parameters (#1100) * removed query parameters annotations (#1101) * do not propagate transpose effect when introducing a transpose operation, maintain expression structure (#1102) * changed RediSearch version to 1.8 (#1103) * changed RediSearch version to 1.8 * added flag for redisearch GC * moved env var setting to memcheck script * Update memcheck.sh * fixed misplaced env var setting * fixed bad command format * flag in circle ci instead of makefile Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * docker file for centos (#1104) * version bump Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: Sceat <11330271+Sceat@users.noreply.github.com> Co-authored-by: Itamar Haber <itamar@redislabs.com> Co-authored-by: Ariel Shtul <ariel.shtul@redislabs.com> Co-authored-by: Timothy Rule <34501912+trulede@users.noreply.github.com> Co-authored-by: Rafi Einstein <rafi@redislabs.com> Co-authored-by: Rafi Einstein <raffapen@outlook.com> Co-authored-by: Christoph Zimmermann <40485189+chrisAtRedis@users.noreply.github.com>

* Enable TCK tests * Introduce Optional and Apply ops * Modify mock AST logic * Emit error on queries beginning with OPTIONAL MATCH * Return null on property accesses of null graph entity * Disallow OPTIONAL MATCH...MATCH queries * Fix OPTIONAL filter placement * Enable TCK tests * NULL handling for path functions * NULL handling for GraphEntity and list functions * WIP improve mock AST logic * Add flow tests * Improve AST mock logic * Error handling for SET and CREATE on null entities * Record_Get refactor * Test null handling * Minor cleanup * Add documentation * Simplify toPath null handling * Improve comments * Allow OPTIONAL MATCH as first clause * Simplify null-checking logic in create ops * Use branch of Python client for testing * PR fixes * PR fixes * Remove Record_GetScalar interface * PR fixes * PR fixes * Add demo query for OPTIONAL MATCH * Use standard Python client for automation * Emit all columns as SIValues in compact formatter * Improve flow test for null entities in first result Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com>

jeffreylovitz added 22 commits March 30, 2020 14:21

Enable TCK tests

a5b99a8

Introduce Optional and Apply ops

5325ad1

Modify mock AST logic

ec9371a

Emit error on queries beginning with OPTIONAL MATCH

f95e5ac

Return null on property accesses of null graph entity

6330061

Disallow OPTIONAL MATCH...MATCH queries

85e7368

Fix OPTIONAL filter placement

d2d7a65

Enable TCK tests

67ae023

NULL handling for path functions

96af9a1

NULL handling for GraphEntity and list functions

cf789b9

WIP improve mock AST logic

d8e58a2

Add flow tests

7906473

Improve AST mock logic

a6332fe

Error handling for SET and CREATE on null entities

b3843c1

Record_Get refactor

8117f07

Test null handling

59083af

Minor cleanup

24cc90a

Add documentation

98f0574

Simplify toPath null handling

0cbd1a6

Improve comments

c312e9a

Allow OPTIONAL MATCH as first clause

de7a126

Simplify null-checking logic in create ops

7defbfd

jeffreylovitz added the cypher label Mar 30, 2020

jeffreylovitz added this to the RedisGraph 2.2 milestone Mar 30, 2020

jeffreylovitz requested review from swilly22 and DvirDukhan March 30, 2020 20:51

jeffreylovitz self-assigned this Mar 30, 2020

Use branch of Python client for testing

61f4965

swilly22 requested changes Mar 31, 2020

View reviewed changes

DvirDukhan reviewed Mar 31, 2020

View reviewed changes

PR fixes

a55e632

swilly22 requested changes Apr 2, 2020

View reviewed changes

jeffreylovitz added 3 commits April 2, 2020 17:50

PR fixes

e9d78ad

Remove Record_GetScalar interface

b7a9884

PR fixes

c2c3145

swilly22 requested changes Apr 3, 2020

View reviewed changes

jeffreylovitz added 2 commits April 3, 2020 09:25

PR fixes

fc0ed1a

Add demo query for OPTIONAL MATCH

2f98977

swilly22 previously approved these changes Apr 3, 2020

View reviewed changes

swilly22 and others added 2 commits April 3, 2020 17:56

Merge branch 'master' into optional-match

e240158

Use standard Python client for automation

a993784

jeffreylovitz dismissed swilly22’s stale review via a993784 April 3, 2020 15:06

jeffreylovitz added 2 commits April 6, 2020 13:00

Emit all columns as SIValues in compact formatter

38ac976

Improve flow test for null entities in first result

732be69

swilly22 approved these changes Apr 6, 2020

View reviewed changes

swilly22 merged commit dc2e23e into master Apr 6, 2020

swilly22 deleted the optional-match branch April 6, 2020 18:26

swilly22 mentioned this pull request Apr 6, 2020

Support for OPTIONAL and DETACH keyword #158

Closed


		In this case, `w.department` and `ID` will be returned if the OPTIONAL MATCH was successful, and will be null otherwise.

		Clauses like SET, CREATE, and DELETE will ignore null inputs and perform the expected updates on real inputs. One exception to this is that attempting to create a relation with a null endpoint will cause an error:

		/* If Optional's child produces records, Optional emits them without modification.
		* If the child produces no records, Optional emits an empty Record exactly once. */

Optional match #1043

Optional match #1043

Conversation

jeffreylovitz commented Mar 30, 2020

swilly22 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swilly22 left a comment

Choose a reason for hiding this comment