Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source: MongoDB - Milliseconds missing in Cursor - Incremental change capture. Source field datatype is - Date #8290

Closed
prconstruct opened this issue Nov 29, 2021 · 0 comments · Fixed by #8306

Comments

@prconstruct
Copy link

prconstruct commented Nov 29, 2021

Enviroment

  • Airbyte version: 0.32.6-alpha
  • OS Version / Instance: AWS EC2
  • Deployment: Docker
  • Source Connector and version: airbyte/source-mongodb-v2:0.1.7
  • Destination Connector and version: S3 - JSONL
  • Severity: Critical
  • Step where error happened: Sync job

Current Behavior

Incremental change capture (cursor) is missing milliseconds. Due to this same records are being captured repeatedly.
For example, Lets assume, I have chosen "UpdatedAt" field as cursor. The last processed record "UpdatedAt" is "2021-11-01 10:59:01.567Z". But the cursor is being captured as "2021-11-01 10:59:01Z". Eventhough there is no changes done at the source, in the next run, the same set of record is captured again because its checking for the records GREATER THAN "2021-11-01 10:59:01Z" so obviously the record which has timestamp "2021-11-01 10:59:01.567Z" is processed again.
Note - The datatype of "updatedAt" field is "Date

Expected Behavior

The cursor should capture the exact data with milliseconds (in the above example case, it should capture the cursor as "2021-11-01 10:59:01.567Z")

Logs

LOG

2021-11-29 07:16:51 INFO () ConfigRepository(updateConnectionState):518 - Updating connection 745bb9ff-7844-47f8-bfa1-c0fda91bf8aa state: io.airbyte.config.State@283707aa[state={"cdc":false,"streams":[{"stream_name":"airbyte_evaluation","stream_namespace":"dev-payconstruct-copy","cursor_field":["updatedAt"],"cursor":"2021-09-08T12:15:37Z"}]}]

Steps to Reproduce

  1. Create a collection and Insert some records in MongoDB. Make sure record has a timestamp field that will be used as a cursor and also the timestamp has microseconds.
  2. Configure source (MongoDb), destination(S3), and connector as Incremental Append. selected a timestamp field as cursor.
  3. Run the connector or run docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/{workspace} --network host --log-driver none airbyte/source-mongodb-v2:0.1.7 read --config source_config.json --catalog source_catalog.json
  4. Rerun the connection again without making any changes in source. This should process the record again.

Are you willing to submit a PR?

Since I'm not proficient in Java, I'm unable to figure the issue in code.

logs-78-0.txt
image (3)
image (2)
image (1)
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

5 participants