Skip to content

Feature/stream name root table canonicalization#131

Merged
AlexanderMann merged 9 commits intodatamill-co:masterfrom
AlexanderMann:feature/stream-name--root-table--canonicalization
Aug 20, 2019
Merged

Feature/stream name root table canonicalization#131
AlexanderMann merged 9 commits intodatamill-co:masterfrom
AlexanderMann:feature/stream-name--root-table--canonicalization

Conversation

@AlexanderMann
Copy link
Copy Markdown
Collaborator

@AlexanderMann AlexanderMann commented Jul 30, 2019

Motivation

#102

  • Provide logic which works in target-redshift as well as target-postgres
    • works across all versions
  • Migrates existing remote databases seamlessly
  • Existing tests remain untouched

Notes

This will break instances where multiple versions of target-postgres are trying to upsert different streams (taps) to the same remote.

I'm unsure as to whether we want to support this use case.

ie, as a user, I have a long running tap persisting data to postgres db A, via target-postgres v0.1.8. I also want to be able to run the latest target-postgres v0.1.X to persist another tap to db A.

Suggested Musical Pairing

https://soundcloud.com/snottynoserezkids/sets/snotty-nose-rez-kids-lp

@AlexanderMann AlexanderMann force-pushed the feature/stream-name--root-table--canonicalization branch from 3f8c5dc to 38baae4 Compare July 31, 2019 22:43
@AlexanderMann
Copy link
Copy Markdown
Collaborator Author

Of note: I realized recently that we assume all the time that there is only one target running per db all the time. This is due to us using transactions on the db, so something should just fail quickly if there's a conflict etc.

@AlexanderMann AlexanderMann merged commit 24e9904 into datamill-co:master Aug 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants