Soft Delete bug

[Tang8330]

Context

Transfer leverages an in-memory database that is key-value based.

The key being the primary key(s) of the table and the value is the latest row as we leverage Kafka for ordering guarantee.

Problem

The problem is that if there are 2 events within one flush cycle say:

• t0, event was created for id = 1
• t1, event was deleted for id = 1

When we flush, we'll only have the latest deleted row. Our merge logic for handling soft deletes was incorrect as it wrongfully assumed that the row should be in the target table.

MERGE INTO foo.public."bar" tgt USING ( foo.public."bar" ) AS stg ON tgt."id" = stg."id" AND tgt."user_id" = stg."user_id"
WHEN MATCHED THEN UPDATE SET "id"=stg."id","user_id"=stg."user_id","__ARTIE_DELETE"=stg."__ARTIE_DELETE","__ARTIE_UPDATED_AT"=stg."__ARTIE_UPDATED_AT"
WHEN NOT MATCHED AND IFNULL(foo.__ARTIE_DELETE, false) = false THEN INSERT ("id","user_id","__ARTIE_DELETE","__ARTIE_UPDATED_AT") VALUES (stg."id",stg."user_id",stg."__ARTIE_DELETE",stg."__ARTIE_UPDATED_AT");

We fixed this by changing WHEN NOT MATCHED AND IFNULL(foo.__ARTIE_DELETE, false) = false to be WHEN NOT MATCHED

[nathan-artie]

Print statement.