-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add normalization to destination definition and actor definition table #18300
Add normalization to destination definition and actor definition table #18300
Conversation
…ags to the destination_definition.yaml and added information about normalization and DBT to the ACTOR_DEFINITION table
…18230_update_StandardDestinationDefinition
@evantahler @tuliren @pedroslopez, I plan to completely close the first step from the document "Decouple Normalization from Platform" with this pull request. In my opinion, this is all that should be done in the first step, but I could be wrong and miss something. If I've done something wrong, please let me know and I'll fix it. |
Hey @andriikorotkov could you please create an issue related to this PR?
As I'm not very familiar with the |
@alafanechere, yes, you are right, I am trying to do this exact part from our documentation (1 step). Also, I have already created subtasks for each of the steps. Also, I seem to have added this pull request to the right task. Link to the issue for the first step - 18230 Link to the issue that I turned into a small epic - 17915 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for providing more description.
I think you need to format your branch to follow our linting rules:
./gradlew format
. (Please install pre-commit to avoid these problem in the future: pre-commit intall
)
Moreover the platform build is failing with this error:
org.opentest4j.AssertionFailedError: expected: <0.40.12.001> but was: <0.40.15.001>
How did you pick the version the migration will be run for?
airbyte-config/config-models/src/main/resources/types/StandardDestinationDefinition.yaml
Show resolved
Hide resolved
…18230_update_StandardDestinationDefinition
Adding @colesnodgrass and @jdpgrailsdev from the platform team for review as well |
airbyte-config/config-models/src/main/resources/types/StandardDestinationDefinition.yaml
Show resolved
Hide resolved
…18230_update_StandardDestinationDefinition
…18230_update_StandardDestinationDefinition
* master: (69 commits) 🪟 🐛 Fix wrong geography dropdown type #19021 SAT: basic read on full catalog when `test_strictness_level == high` (#18937) Unhide DynamoDB destination (#18994) Fixed tests for destination connectors (#19007) 🐛 Source Facebook Marketing: handle FacebookBadObjectError (#18971) Edit multi-cloud docs (#18972) 🪟 🎉 Load credits consumption separate (#18986) Bmoric/extract source api (#18944) Migrating InvalidCursorException -> ConfigErrorException (#18995) 🪟 🎨 Fix banner link color (#18978) Handling configuration exceptions in IntegrationRunner (#18989) Add new workspace api endpoint (#18983) Add normalization to destination definition and actor definition table (#18300) Fix oauth controller (#18981) Fix migration dev center schema dump by run db-specific initialization script (#18984) fix master build failure (#18982) cleanup: delete debezium 1-4-2 module (#18733) Remove unused job persistence methods. (#18952) Hash filenames of extracted CSS (#18976) Fix typo in source code comment DataDaog ==> Datadog (#18911) ...
* master: (73 commits) 🪟 🐛 Fix wrong geography dropdown type #19021 SAT: basic read on full catalog when `test_strictness_level == high` (#18937) Unhide DynamoDB destination (#18994) Fixed tests for destination connectors (#19007) 🐛 Source Facebook Marketing: handle FacebookBadObjectError (#18971) Edit multi-cloud docs (#18972) 🪟 🎉 Load credits consumption separate (#18986) Bmoric/extract source api (#18944) Migrating InvalidCursorException -> ConfigErrorException (#18995) 🪟 🎨 Fix banner link color (#18978) Handling configuration exceptions in IntegrationRunner (#18989) Add new workspace api endpoint (#18983) Add normalization to destination definition and actor definition table (#18300) Fix oauth controller (#18981) Fix migration dev center schema dump by run db-specific initialization script (#18984) fix master build failure (#18982) cleanup: delete debezium 1-4-2 module (#18733) Remove unused job persistence methods. (#18952) Hash filenames of extracted CSS (#18976) Fix typo in source code comment DataDaog ==> Datadog (#18911) ...
What
updated
StandardDestinationDefinition.yaml
, added normalization and tags to thedestination_definition.yaml
and added information about normalization and DBT to the ACTOR_DEFINITION tableHow
I added the fields we need for DBT and Normalization in
StandardDestinationDefinition.yaml
. Also, I expanded theACTOR_DEFINITION
table a bit, in order to store the values of destinations normalization and DBT where needed (the new fields can be null, since theACTOR_DEFINITION
table also stores information about sources and destinations that do not support normalization). Since the path by which the migration file was created has already been added to gradle, during the build of the platform, the new migration will be performed, and the fields will be added to the table.Recommended reading order
StandardDestinationDefinition.yaml
V0_40_15_001__AddActorDefinitionNormalizationAndDbtColumns.java
ConfigWriter.java
🚨 User Impact 🚨
Currently, it does not add any impact to the platform. But in the future, we will use values for normalization and DBT fields from the definition file.