You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Source Connector and version: airbyte/source-mongodb-v2:0.1.7
Destination Connector and version: S3 - JSONL
Severity: Critical
Step where error happened: Sync job
Current Behavior
Incremental change capture (cursor) is missing milliseconds. Due to this same records are being captured repeatedly.
For example, Lets assume, I have chosen "UpdatedAt" field as cursor. The last processed record "UpdatedAt" is "2021-11-01 10:59:01.567Z". But the cursor is being captured as "2021-11-01 10:59:01Z". Eventhough there is no changes done at the source, in the next run, the same set of record is captured again because its checking for the records GREATER THAN "2021-11-01 10:59:01Z" so obviously the record which has timestamp "2021-11-01 10:59:01.567Z" is processed again. Note - The datatype of "updatedAt" field is "Date
Expected Behavior
The cursor should capture the exact data with milliseconds (in the above example case, it should capture the cursor as "2021-11-01 10:59:01.567Z")
Create a collection and Insert some records in MongoDB. Make sure record has a timestamp field that will be used as a cursor and also the timestamp has microseconds.
Configure source (MongoDb), destination(S3), and connector as Incremental Append. selected a timestamp field as cursor.
Run the connector or run docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/{workspace} --network host --log-driver none airbyte/source-mongodb-v2:0.1.7 read --config source_config.json --catalog source_catalog.json
Rerun the connection again without making any changes in source. This should process the record again.
Are you willing to submit a PR?
Since I'm not proficient in Java, I'm unable to figure the issue in code.
Enviroment
Current Behavior
Incremental change capture (cursor) is missing milliseconds. Due to this same records are being captured repeatedly.
For example, Lets assume, I have chosen "UpdatedAt" field as cursor. The last processed record "UpdatedAt" is "2021-11-01 10:59:01.567Z". But the cursor is being captured as "2021-11-01 10:59:01Z". Eventhough there is no changes done at the source, in the next run, the same set of record is captured again because its checking for the records GREATER THAN "2021-11-01 10:59:01Z" so obviously the record which has timestamp "2021-11-01 10:59:01.567Z" is processed again.
Note - The datatype of "updatedAt" field is "Date
Expected Behavior
The cursor should capture the exact data with milliseconds (in the above example case, it should capture the cursor as "2021-11-01 10:59:01.567Z")
Logs
LOG
Steps to Reproduce
Are you willing to submit a PR?
Since I'm not proficient in Java, I'm unable to figure the issue in code.
logs-78-0.txt
The text was updated successfully, but these errors were encountered: