Stream Enrich: ensure a one-to-one relationship between sink and record processor

[asgergb]

After upgrading to R102 from R96 following the Upgrade Guide, I see duplicated events in the enriched stream in Kinesis.

The image below which shows the number of events in the raw web stream (prior to enrichment), the good enriched stream and the bad enriched stream. As you can see there are many more events in the enriched good stream than the raw stream after deploying R102 at 9:40. I redeployed R102 at 10:10 and then rolled back to R96 at 10:20. Looking at the enriched good stream, around half or so of the events are duplicated, that is, two identical events are on the stream.

Due to #3744, I am unable to test R103, and I have not been able to find the cause. Do you have any idea what's happening?

[BenFradet]

hey @asgergb , could you expose your issue in our discourse and provide as much info as possible (number of shards of each stream, number of instances for stream enrich, etc) so we can try to reproduce?

[asgergb]

Sure, I'll do that as soon as possible. We have a holiday coming up but I'll try to get around to it.

[asgergb]

FYI I have tried the official artifacts for R103 (collector 0.13.0 and enricher 0.16.0) as part of #3744, and the result is the same with roughly half of all events being duplicated.

[asgergb]

@BenFradet Here you go: https://discourse.snowplowanalytics.com/t/stream-enrich-duplicated-enriched-events-in-r103-3745/1986

[BenFradet]

The problem was that the same sink was reused across kcl's IRecordProcessors and so the same sink would flush as many times as there were shards.