Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed gl_journal_entries replication key #56

Conversation

DownstreamDataTeam
Copy link
Contributor

@DownstreamDataTeam DownstreamDataTeam commented Oct 22, 2021

Description of change

Changed replication_keys and bookmark_field for gl_journal_entries stream from booking_date to creation_date.
The problem that this change solves is that, currently, we are missing journal entries records when transactions are reversed.

A test that was performed (without code changes):

  1. set start_date to current date
  2. reversed a transaction with a booking date in the past
  3. synced the gl_journal_entries stream records.

The expected result: at least the reversed transaction records.
The actual result: didn't get the records for the reversed transaction.

After the change from booking_date to creation_date, the reversed transactions were synced.

Here are some json snippets ("start_date": "2021-10-18T00:00:00.000000Z"):
Result before code changes

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31986", "creation_date": "2021-10-17T21:25:32.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "295.75", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31987", "creation_date": "2021-10-17T21:25:32.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "295.75", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31990", "creation_date": "2021-10-17T21:26:03.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "277.78", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "31991", "creation_date": "2021-10-17T21:26:03.000000Z", "entry_date": "2021-10-18T00:00:00.000000Z", "amount": "277.78", "booking_date": "2021-10-18T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:49:40.267352Z"}

Result after code changes

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32094", "creation_date": "2021-10-18T09:38:40.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "38.33", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32095", "creation_date": "2021-10-18T09:38:40.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "1.67", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32096", "creation_date": "2021-10-18T09:38:40.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "40", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32097", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-13T00:00:00.000000Z", "amount": "20", "booking_date": "2021-10-13T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32098", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-13T00:00:00.000000Z", "amount": "18.33", "booking_date": "2021-10-13T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32099", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-13T00:00:00.000000Z", "amount": "1.67", "booking_date": "2021-10-13T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32100", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "40", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

{"type": "RECORD", "stream": "gl_journal_entries", "record": {"entry_id": "32101", "creation_date": "2021-10-18T09:38:41.000000Z", "entry_date": "2021-10-15T00:00:00.000000Z", "amount": "40", "booking_date": "2021-10-15T00:00:00.000000Z"}, "time_extracted": "2021-10-18T09:48:41.300660Z"}

Rollback steps

  • revert this branch

@cmerrick
Copy link
Contributor

Hi @DownstreamDataTeam, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

@gvillafanetapia
Copy link

This looks great we have the exact same issue!

@leslievandemark leslievandemark merged commit b3a8962 into singer-io:master Nov 3, 2021
@singer-bot
Copy link

You did it @DownstreamDataTeam!

Thank you for signing the Singer Contribution License Agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants