[Bug][mongo] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U(or -U)#2760
Conversation
|
@Jiabao-Sun , @Shawn-Hx , CC |
|
+U and +I should have the same processing in the external system. |
@Jiabao-Sun , this +U is without same key data(+I) before, which is not required the changelog semantic. What happens if the downstream need the data before when process +U record? I wonder whether a +U (without +I before) will cause other problems in streaming compute or not? @leonardBang , CC , WDYT? |
|
@Jiabao-Sun , moreover, if |
-U and +U contains the same key, it should be covered by +U ? |
@Jiabao-Sun , |
|
I agree with @loserwang1024. Logically, data before highwatermark are snapshot data, which should be seen as +I. Otherwise the downstream operators may see a +U/-U record without +I before. I'm not sure whether this behavior may cause data incorrectness or not, but using +I is correct with no doubt. @Jiabao-Sun |
|
OK, @loserwang1024 could you help add some tests for this? |
|
@Jiabao-Sun , thanks a lot. It's not easy to add test now. |
| case INSERT: | ||
| case UPDATE: | ||
| case REPLACE: | ||
| value.put(MongoDBEnvelope.OPERATION_TYPE_FIELD, INSERT.getValue()); |
There was a problem hiding this comment.
| value.put(MongoDBEnvelope.OPERATION_TYPE_FIELD, INSERT.getValue()); | |
| value.put(MongoDBEnvelope.OPERATION_TYPE_FIELD, OperationType.INSERT.getValue()); |
There was a problem hiding this comment.
Thanks a lot , I will do it
a212811 to
6162be5
Compare
6162be5 to
2e10978
Compare
Fix #2759