Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed gl_journal_entries replication key #56


Copy link

@DownstreamDataTeam DownstreamDataTeam commented Oct 22, 2021

Description of change

Changed replication_keys and bookmark_field for gl_journal_entries stream from booking_date to creation_date.
The problem that this change solves is that, currently, we are missing journal entries records when transactions are reversed.

A test that was performed (without code changes):

  1. set start_date to current date
  2. reversed a transaction with a booking date in the past
  3. synced the gl_journal_entries stream records.

The expected result: at least the reversed transaction records.
The actual result: didn't get the records for the reversed transaction.

After the change from booking_date to creation_date, the reversed transactions were synced.

Here are some json snippets ("start_date": "2021-10-18T00:00:00.000000Z"):
Result before code changes

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31986", "creation_date": "2021-10-17T21:25:32.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "295.75", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31987", "creation_date": "2021-10-17T21:25:32.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "295.75", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31990", "creation_date": "2021-10-17T21:26:03.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "277.78", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31991", "creation_date": "2021-10-17T21:26:03.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "277.78", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

Result after code changes

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32094", "creation_date": "2021-10-18T09:38:40.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "38.33", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32095", "creation_date": "2021-10-18T09:38:40.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "1.67", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32096", "creation_date": "2021-10-18T09:38:40.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "40", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32097", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-13T00:00:00.000000Z", "amount": "20", "booking_date": "2021-10-13T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32098", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-13T00:00:00.000000Z", "amount": "18.33", "booking_date": "2021-10-13T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32099", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-13T00:00:00.000000Z", "amount": "1.67", "booking_date": "2021-10-13T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32100", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "40", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32101", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "40", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

Rollback steps

  • revert this branch

Copy link

Hi @DownstreamDataTeam, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

Copy link

This looks great we have the exact same issue!

@leslievandemark leslievandemark merged commit b3a8962 into singer-io:master Nov 3, 2021
Copy link

You did it @DownstreamDataTeam!

Thank you for signing the Singer Contribution License Agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

6 participants