Set the record offset (specifically 'snapshot' and 'snapshot_completed' fields) explicitly based on the incoming source record for the SQL server source.#149
Conversation
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
| SqlServerOffset dmlRecordOffset = new SqlServerOffset(sourceRecord.sourceOffset(), ddlEventSent); | ||
| Long ingestTime = val.get("ts_ms"); | ||
| DMLEvent.Builder dmlBuilder = DMLEvent.builder() | ||
| .setOffset(latestOffset) |
There was a problem hiding this comment.
latestOffset before fix didn't have updated ddlEventSent set, when only 1 record was sent as a part of snapshot and pipeline was stopped after.
| // if (!sqlServerOffset.isSnapshot() && sqlServerOffset.isBeforeOrAt(latestOffset)) { | ||
| // LOG.debug("Got duplicated event {} ", sourceRecord); | ||
| // return; | ||
| // } |
There was a problem hiding this comment.
uncomment the codes?
| // send the ddl events only if we see the table at the first time | ||
| // Note: the delta app itself have prevented adding CREATE_TABLE operation into DDL blacklist for all the tables. | ||
| if (!ddlEventSent.contains(sourceTableId)) { | ||
| ddlEventSent.add(sourceTableId); |
There was a problem hiding this comment.
adding table id here may cause some problem , because there are three DDL event sent below :
- DROP TABLE
- CREATE DATABSE
- CREATE TALBE
what if target failed before committing offset of 3 and after committing offset of 1 and 2.
since 1 and 2 already has the offset containing the table id, we won't generate CREATE TABLE event again after resuming.
There was a problem hiding this comment.
my suggestion would be adding it to the offset of first dml event.
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
…d' fields) explicitly based on the incoming source record for the SQL server source.
sagarkapare
force-pushed
the
fix/set-offset-correctly
branch
from
6634a47 to
798106e
Compare
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
2 participants
Also made
SqlServerOffsetclass immutable.Following specific scenario was failing without the fix:
Pipeline generated DDL events (DROP and CREATE tables) again after step 4 because
ddlEventSentfield was not persisted at the end of step 1.