Normalization: solve conflict when stream and field have same name #4557

marcosmarxm · 2021-07-06T00:08:42Z

What

How

Add a check when parent stream_name is equal to stream_name.
@ChristopheDuong this only solve for a 1 level nested conflict. Do you think should be something random to handle deeper conflicts?

Pre-merge Checklist

Expand the checklist which is relevant for this PR.

Connector checklist

Connector Generator checklist

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

marcosmarxm · 2021-07-06T00:10:50Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1002686506
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1002686506

ChristopheDuong · 2021-07-06T08:59:10Z

airbyte-integrations/bases/base-normalization/integration_tests/test_normalization.py

@@ -74,7 +74,8 @@ def setup_test_path(request):
        ]
    ),
 )
-@pytest.mark.parametrize("destination_type", list(DestinationType))
+# @pytest.mark.parametrize("destination_type", list(DestinationType))
+@pytest.mark.parametrize("destination_type", [DestinationType.POSTGRES])


remember to revert this

marcosmarxm · 2021-07-07T03:31:48Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1006670650
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1006670650

marcosmarxm · 2021-07-07T03:32:31Z

@ChristopheDuong I added logic using json_path but look mysql, bigquery and snowflake failed because the json extraction with same name...

marcosmarxm · 2021-07-07T13:56:29Z

.../generated/airbyte_ctes/test_normalization/conflict_stream_name_conflict_stream_name_ab1.sql

+-- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema
+select
+    _airbyte_conflict_stream_name_hashid,
+    {{ json_extract('conflict_stream_name', ['conflict_stream_name']) }} as conflict_stream_name,


@ChristopheDuong here

{"conflict_name": {"conflict_name": "hi"}}

if somehow the macro would translate into something like json_extract('conflict_stream_name', ['conflict_stream_name', 'conflict_stream_name'])

maybe that would make it less ambiguous for the SQL engine (Destination)?

marcosmarxm · 2021-07-21T21:25:42Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1054076661
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1054076661

marcosmarxm · 2021-07-21T21:29:13Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1054085229
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1054085229

ChristopheDuong · 2021-07-22T15:36:40Z

FYI I've been looking at fixing normalization integration tests here: #4910

(and you might need to merge master as I also included some other fixes recently too, on top of all the conflicts this generated...)

marcosmarxm · 2021-07-28T03:06:45Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1073474397
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1073474397

marcosmarxm · 2021-07-28T04:01:37Z

@ChristopheDuong mysql is getting a cycle and I'm having problems to run locally the integration test for mysql.

Encountered an error:
Found a cycle: model.airbyte_utils.conflict_stream_name_conflict_stream_name --> model.airbyte_utils.conflict_stream_name___flict_stream_name_ab1 --> model.airbyte_utils.conflict_stream_name___flict_stream_name_ab2 --> model.airbyte_utils.conflict_stream_name___flict_stream_name_ab3

Redshift failed and locally is not working too, the problem I'm 100% it already worked before :(

ChristopheDuong · 2021-07-28T09:20:40Z

Redshift failed and locally is not working too, the problem I'm 100% it already worked before :(

Yes, redshift is broken for me too

The test is failing on writing raw JSON records to redshift using only the destination-redshift connector... so it is not even starting to run anything about normalization yet.

ChristopheDuong · 2021-07-28T09:21:20Z

/test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1074350881
❌ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1074350881

ChristopheDuong · 2021-07-28T14:35:23Z

...te-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py

@@ -322,7 +323,7 @@ def generate_json_parsing_model(self, from_table: str, column_names: Dict[str, T
    {{ field }},
  {%- endfor %}
    _airbyte_emitted_at
-from {{ from_table }}
+from {{ from_table }} as {{ table_alias }}


YOu need to introduce table_alias in only the extract_json cte not all of them

marcosmarxm · 2021-07-28T16:10:18Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1075570298
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1075570298

marcosmarxm · 2021-08-04T15:30:16Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1098266840
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1098266840

.../test_primary_key_streams/final/airbyte_ctes/test_normalization/conflict_stream_name_ab2.sql

...es/test_normalization/conflict_stream_name_conflict_stream_name_conflict_stream_name_ab1.sql

airbyte-integrations/bases/base-normalization/integration_tests/test_normalization.py

...ons/bases/base-normalization/normalization/transform_catalog/destination_name_transformer.py

...te-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py

marcosmarxm · 2021-08-09T22:18:39Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1114533773
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1114533773

ChristopheDuong · 2021-08-10T14:12:44Z

...zation/integration_tests/resources/test_primary_key_streams/dbt_schema_tests/schema_test.yml

+#    tests:
+#      - dbt_utils.expression_is_true:
+#          expression: "double_array_data is not null"
+#      - dbt_utils.expression_is_true:
+#          expression: "DATA is not null"
+#      - dbt_utils.expression_is_true:
+#          expression: "\"column`_'with\"\"_quotes\" is not null"


can you open a new issue to look closer into this later? thanks

marcosmarxm · 2021-08-11T16:11:34Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1120781700
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1120781700

marcosmarxm · 2021-08-11T17:05:34Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1120946518
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1120946518

marcosmarxm · 2021-08-11T17:39:36Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1121048854
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1121048854

ChristopheDuong · 2021-09-14T09:29:32Z

airbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.py

@@ -115,6 +118,7 @@ def setup_mysql_db(self):
        ]
        print("Executing: ", " ".join(commands))
        subprocess.call(commands)
+        time.sleep(120)


This should not be here, it makes the test wait for two minutes before doing anything

Sorry Chris! I add this because in my Ubuntu Setup the mysql and postgres took longer to start. I created this: #6091

solve conflict when stream and field have same name

9378160

marcosmarxm requested a review from ChristopheDuong July 6, 2021 00:08

github-actions bot added the normalization label Jul 6, 2021

ChristopheDuong reviewed Jul 6, 2021

View reviewed changes

marcosmarxm added 2 commits July 7, 2021 00:27

add logic to handle conflict

65a50a3

change files

d9281a6

marcosmarxm commented Jul 7, 2021

View reviewed changes

marcosmarxm added 2 commits July 8, 2021 18:04

Merge branch 'master' into marcosmarxm/conflict-hashid-normalization

9c25632

change json_extract functions

e568fd4

json_operations

496ed7c

merge master

7f51815

add normalization files

9aab5f8

test integration mysql

587c582

ChristopheDuong reviewed Jul 28, 2021

View reviewed changes

remove table_alias

ffa41fd

marcosmarxm added 3 commits August 4, 2021 13:11

mysql run

eead1c1

json ops

e9bd926

Merge branch 'master' into marcosmarxm/conflict-hashid-normalization

fa62a78

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

.../test_primary_key_streams/final/airbyte_ctes/test_normalization/conflict_stream_name_ab2.sql Outdated Show resolved Hide resolved

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

...es/test_normalization/conflict_stream_name_conflict_stream_name_conflict_stream_name_ab1.sql Show resolved Hide resolved

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

airbyte-integrations/bases/base-normalization/integration_tests/test_normalization.py Outdated Show resolved Hide resolved

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

...ons/bases/base-normalization/normalization/transform_catalog/destination_name_transformer.py Outdated Show resolved Hide resolved

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

...ons/bases/base-normalization/normalization/transform_catalog/destination_name_transformer.py Outdated Show resolved Hide resolved

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

...te-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py Show resolved Hide resolved

ChristopheDuong reviewed Aug 9, 2021

View reviewed changes

...te-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py Show resolved Hide resolved

add tests for scalar and arrays

5b1fc0b

marcosmarxm requested a review from ChristopheDuong August 9, 2021 22:18

jrhizor temporarily deployed to more-secrets August 9, 2021 22:21 Inactive

ChristopheDuong approved these changes Aug 10, 2021

View reviewed changes

ChristopheDuong reviewed Aug 10, 2021

View reviewed changes

subodh1810 mentioned this pull request Aug 10, 2021

introduce implementation for date-time support in normalization #5180

Merged

36 tasks

Merge branch 'master' into marcosmarxm/conflict-hashid-normalization

9f79d09

jrhizor temporarily deployed to more-secrets August 11, 2021 16:14 Inactive

marcosmarxm added 2 commits August 11, 2021 13:39

add sql files

4042439

bump normalization version

89d2d8b

github-actions bot added area/documentation Improvements or additions to documentation area/worker Related to worker labels Aug 11, 2021

jrhizor temporarily deployed to more-secrets August 11, 2021 17:08 Inactive

format

57bbce6

jrhizor temporarily deployed to more-secrets August 11, 2021 17:42 Inactive

Merge branch 'master' into marcosmarxm/conflict-hashid-normalization

32e1518

marcosmarxm merged commit e4fe62f into master Aug 11, 2021

marcosmarxm deleted the marcosmarxm/conflict-hashid-normalization branch August 11, 2021 23:18

ChristopheDuong reviewed Sep 14, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Normalization: solve conflict when stream and field have same name #4557

Normalization: solve conflict when stream and field have same name #4557

marcosmarxm commented Jul 6, 2021

marcosmarxm commented Jul 6, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Jul 6, 2021

marcosmarxm commented Jul 7, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 7, 2021

marcosmarxm Jul 7, 2021

marcosmarxm Jul 7, 2021

ChristopheDuong Jul 7, 2021

marcosmarxm commented Jul 21, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 21, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Jul 22, 2021

marcosmarxm commented Jul 28, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 28, 2021

ChristopheDuong commented Jul 28, 2021

ChristopheDuong commented Jul 28, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Jul 28, 2021

marcosmarxm commented Jul 28, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 4, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 9, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Aug 10, 2021

marcosmarxm commented Aug 11, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 11, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 11, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Sep 14, 2021

marcosmarxm Sep 15, 2021

Normalization: solve conflict when stream and field have same name #4557

Normalization: solve conflict when stream and field have same name #4557

Conversation

marcosmarxm commented Jul 6, 2021

What

How

Recommended reading order

Pre-merge Checklist

marcosmarxm commented Jul 6, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

marcosmarxm commented Jul 7, 2021 • edited by github-actions bot Loading

marcosmarxm commented Jul 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosmarxm commented Jul 21, 2021 • edited by github-actions bot Loading

marcosmarxm commented Jul 21, 2021 • edited by github-actions bot Loading

ChristopheDuong commented Jul 22, 2021

marcosmarxm commented Jul 28, 2021 • edited by github-actions bot Loading

marcosmarxm commented Jul 28, 2021

ChristopheDuong commented Jul 28, 2021

ChristopheDuong commented Jul 28, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

marcosmarxm commented Jul 28, 2021 • edited by github-actions bot Loading

marcosmarxm commented Aug 4, 2021 • edited by github-actions bot Loading

marcosmarxm commented Aug 9, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

marcosmarxm commented Aug 11, 2021 • edited by github-actions bot Loading

marcosmarxm commented Aug 11, 2021 • edited by github-actions bot Loading

marcosmarxm commented Aug 11, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcosmarxm commented Jul 6, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 7, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 21, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 21, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 28, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Jul 28, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Jul 28, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 4, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 9, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 11, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 11, 2021 •

edited by github-actions bot

Loading

marcosmarxm commented Aug 11, 2021 •

edited by github-actions bot

Loading