Normalization - BigQuery use `json_extract_string_array` for array of simple types #13289

grubberr · 2022-05-27T18:21:18Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

For array extraction use better BigQuery json function

json_extract_array -> json_extract_string_array

it produce correct strings inside array
before:

['"2022-11-22T01:23:45+05:00"', '"9999-12-21T01:23:45-05:00"']

after:

['2022-11-22T01:23:45+05:00', '9999-12-21T01:23:45-05:00']

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-05-27T18:29:22Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2398150643
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2398150643
🐛 https://gradle.com/s/cl3jcrem5rxtk

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_ephemeral.py:54: Destinations DestinationType.MYSQL is not in NORMALIZATION_TEST_TARGET env variable (MYSQL is also skipped)
	 SKIPPED [1] integration_tests/test_normalization.py:75: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:137: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:125: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:75: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:125: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_nested_streams]
	 �[31m============= �[31m�[1m1 failed�[0m, �[32m24 passed�[0m, �[33m9 skipped�[0m�[31m in 2780.68s (0:46:20)�[0m�[31m =============�[0m

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-05-28T15:06:36Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2401405735
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2401405735
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 545    354    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1335    561    58%
Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 545    354    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1335    561    58%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/utils.py                             38      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      5    97%
normalization/transform_catalog/stream_processor.py                 545     42    92%
normalization/transform_catalog/catalog_processor.py                147     12    92%
normalization/transform_catalog/dbt_macro.py                         22      3    86%
normalization/transform_catalog/table_name_registry.py              174     51    71%
normalization/transform_config/transform.py                         159     55    65%
normalization/transform_catalog/transform.py                         61     22    64%
-------------------------------------------------------------------------------------
TOTAL                                                              1335    190    86%

Build Passed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_ephemeral.py:54: Destinations DestinationType.MYSQL is not in NORMALIZATION_TEST_TARGET env variable (MYSQL is also skipped)
	 SKIPPED [1] integration_tests/test_normalization.py:75: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:137: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:125: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:75: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:125: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 �[32m================== �[32m�[1m25 passed�[0m, �[33m9 skipped�[0m�[32m in 2402.39s (0:40:02)�[0m�[32m ==================�[0m
	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_ephemeral.py:54: Destinations DestinationType.MYSQL is not in NORMALIZATION_TEST_TARGET env variable (MYSQL is also skipped)
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:75: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:137: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:125: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:125: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:75: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:134: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 �[32m================== �[32m�[1m25 passed�[0m, �[33m9 skipped�[0m�[32m in 2486.29s (0:41:26)�[0m�[32m ==================�[0m

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

evantahler

Nice work!

Date parsing is evil - please add a test (maybe here) to show that the example from the description really does work:

before:

['"2022-11-22T01:23:45+05:00"', '"9999-12-21T01:23:45-05:00"']

after:

['2022-11-22T01:23:45+05:00', '9999-12-21T01:23:45-05:00']

evantahler · 2022-06-09T16:27:14Z

There may be a similar issue with Postgres as well - #13380

grubberr · 2022-06-09T16:31:14Z

@evantahler, we discussed with @DoNotPanicUA that this PR will be more about:
"correct extraction array of strings FOR native array type of BigQuery"

Currently, base-normalization cannot extract an array of dates, I think we will implement it in the future.

Provided example with dates was not very good, more better example would be:

before (with strange quotes )

['"string1"', '"string2"']

after (without strange quotes)

['string1', 'string2']

about tests, @DoNotPanicUA already wrote a test for this case:

airbyte/airbyte-integrations/connectors/source-bigquery/src/test-integration/java/io/airbyte/integrations/source/bigquery/BigQuerySourceDatatypeTest.java

Line 287 in 1b3fe7e

.addExpectedValues("[\"a\",\"b\"]")

evantahler

Ah, got it. The A/B Test makes sense now that I know the 'date' part isn't included

grubberr · 2022-06-09T16:45:57Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2469874331
❌ Failed to publish bases/base-normalization
❌ Couldn't auto-bump version for bases/base-normalization

DoNotPanicUA · 2022-06-09T17:25:28Z

@evantahler @grubberr
The PR is about the Destination fix.
But also include a small fix for source tests. The destination cases including dates and all other possible values are tested by DAT (Destination Acceptance Test).
https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_messages.txt
Here you can find test data for arrays and checked values.

grubberr · 2022-06-09T17:46:30Z

@DoNotPanicUA ah thank for fix, I provided wrong url
but my main point was that my improvement treat all array contents as strings

we still cannot parse array of dates for example
(it's about BigQuery ARRAY type)

grubberr · 2022-06-09T19:43:49Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2470778574
❌ Failed to publish bases/base-normalization
❌ Couldn't auto-bump version for bases/base-normalization

grubberr · 2022-06-10T04:41:23Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2472776509
❌ Failed to publish bases/base-normalization
❌ Couldn't auto-bump version for bases/base-normalization

grubberr · 2022-06-10T06:21:31Z

I cannot PASS tests because writing to MSSQL failed on network for some reason

Connection is not available, request timed out after 60001ms.

https://github.com/airbytehq/airbyte/runs/6825088834?check_suite_focus=true#step:17:8426

I CAN pass this test on local machine

publishing without tests

grubberr · 2022-06-10T06:22:10Z

/publish connector=bases/base-normalization run-tests=false

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2473156585
❌ Failed to publish bases/base-normalization
❌ Couldn't auto-bump version for bases/base-normalization

evantahler · 2022-06-10T19:18:16Z

Working on fixing normalization publishing here #13693

grubberr · 2022-06-10T19:30:51Z

/publish connector=bases/base-normalization run-tests=false

evantahler · 2022-06-10T19:59:56Z

^ looks like the bot never found and linked the right action :(

grubberr · 2022-06-10T20:02:20Z

/publish connector=bases/base-normalization run-tests=false

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/2477229479
🚀 Successfully published bases/base-normalization
❌ Couldn't auto-bump version for bases/base-normalization

bazarnov · 2022-06-17T12:34:25Z

Hi, guys, this change seems to break DAT test for Redshift, Snowflake, Postgres, since there is no JSON_EXTRACT_ARRAY function in these databases, but the default macro applies it to all other except of BigQuery, please take a look everyone involved into the change.
Here is the Issue created based on all above:
#13875

json_extract_array -> json_extract_string_array

db5c5a3

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added area/documentation Improvements or additions to documentation area/platform issues related to the platform area/worker Related to worker normalization labels May 27, 2022

grubberr temporarily deployed to more-secrets May 27, 2022 18:22 Inactive

basic-normalization.md updated

722a7b5

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets May 27, 2022 18:24 Inactive

grubberr self-assigned this May 27, 2022

grubberr linked an issue May 27, 2022 that may be closed by this pull request

🐛 Destination BigQuery : normalization incorrectly processes arrays #13154

Closed

REF -> ref (revert me)

f2e1aa3

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets May 27, 2022 18:31 Inactive

json_extract_string_array added

3e8211f

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets May 28, 2022 11:16 Inactive

improve checking items.type

5c7fa0e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets May 28, 2022 15:08 Inactive

grubberr added 2 commits May 29, 2022 18:10

Merge branch 'master' into grubberr/13154-base-normalization

251f693

turn on tests supportArrayDataTypeTest

205d3b2

github-actions bot added the area/connectors Connector related issues label May 29, 2022

This comment was marked as outdated.

Sign in to view

grubberr temporarily deployed to more-secrets May 29, 2022 15:14 Inactive

grubberr changed the title ~~Normalization - bugfix bigquery array parsing~~ Normalization - BigQuery use json_extract_string_array for array of simple types Jun 2, 2022

grubberr added 2 commits June 2, 2022 12:26

Merge branch 'master' into grubberr/13154-base-normalization

7ff9d0e

updated basic-normalization.md

bb1863b

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets June 2, 2022 09:31 Inactive

Fix reading of array values

9b87d77

DoNotPanicUA temporarily deployed to more-secrets June 2, 2022 10:48 Inactive

bleonard added the connectors/destination/bigquery label Jun 2, 2022

grubberr temporarily deployed to more-secrets June 9, 2022 03:45 Inactive

Merge branch 'master' into grubberr/13154-base-normalization

1b3fe7e

DoNotPanicUA temporarily deployed to more-secrets June 9, 2022 10:32 Inactive

grubberr requested a review from evantahler June 9, 2022 13:36

evantahler requested changes Jun 9, 2022

View reviewed changes

evantahler approved these changes Jun 9, 2022

View reviewed changes

Merge branch 'master' into grubberr/13154-base-normalization

9bc94ec

grubberr temporarily deployed to more-secrets June 10, 2022 04:37 Inactive

Merge branch 'master' into grubberr/13154-base-normalization

3b29a74

grubberr temporarily deployed to more-secrets June 10, 2022 19:32 Inactive

grubberr merged commit 2daaf5b into master Jun 10, 2022

grubberr deleted the grubberr/13154-base-normalization branch June 10, 2022 20:31

octavia-squidington-iii mentioned this pull request Jun 14, 2022

Bump Airbyte version from 0.39.17-alpha to 0.39.18-alpha #13759

Merged

bazarnov mentioned this pull request Jun 17, 2022

Destination Redshift broken integration tests in the master branch #13875

Closed

grubberr mentioned this pull request Jun 17, 2022

Normalization: Fix incorrect jinja2 macro json_extract_array call #13894

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Normalization - BigQuery use `json_extract_string_array` for array of simple types #13289

Normalization - BigQuery use `json_extract_string_array` for array of simple types #13289

grubberr commented May 27, 2022 •

edited

Loading

grubberr commented May 27, 2022 •

edited by github-actions bot

Loading

grubberr commented May 28, 2022 •

edited by github-actions bot

Loading

This comment was marked as outdated.

evantahler left a comment

evantahler commented Jun 9, 2022

grubberr commented Jun 9, 2022

evantahler left a comment

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jun 9, 2022 •

edited

Loading

grubberr commented Jun 9, 2022 •

edited

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 10, 2022

grubberr commented Jun 10, 2022 •

edited by github-actions bot

Loading

evantahler commented Jun 10, 2022 •

edited

Loading

grubberr commented Jun 10, 2022

evantahler commented Jun 10, 2022

grubberr commented Jun 10, 2022 •

edited by github-actions bot

Loading

bazarnov commented Jun 17, 2022

Normalization - BigQuery use json_extract_string_array for array of simple types #13289

Normalization - BigQuery use json_extract_string_array for array of simple types #13289

Conversation

grubberr commented May 27, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

grubberr commented May 27, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented May 28, 2022 • edited by github-actions bot Loading

Build Passed

This comment was marked as outdated.

evantahler left a comment

Choose a reason for hiding this comment

evantahler commented Jun 9, 2022

grubberr commented Jun 9, 2022

evantahler left a comment

Choose a reason for hiding this comment

grubberr commented Jun 9, 2022 • edited by github-actions bot Loading

DoNotPanicUA commented Jun 9, 2022 • edited Loading

grubberr commented Jun 9, 2022 • edited Loading

grubberr commented Jun 9, 2022 • edited by github-actions bot Loading

grubberr commented Jun 10, 2022 • edited by github-actions bot Loading

grubberr commented Jun 10, 2022

grubberr commented Jun 10, 2022 • edited by github-actions bot Loading

evantahler commented Jun 10, 2022 • edited Loading

grubberr commented Jun 10, 2022

evantahler commented Jun 10, 2022

grubberr commented Jun 10, 2022 • edited by github-actions bot Loading

bazarnov commented Jun 17, 2022

Normalization - BigQuery use `json_extract_string_array` for array of simple types #13289

Normalization - BigQuery use `json_extract_string_array` for array of simple types #13289

grubberr commented May 27, 2022 •

edited

Loading

grubberr commented May 27, 2022 •

edited by github-actions bot

Loading

grubberr commented May 28, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Jun 9, 2022 •

edited

Loading

grubberr commented Jun 9, 2022 •

edited

Loading

grubberr commented Jun 9, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Jun 10, 2022 •

edited by github-actions bot

Loading

evantahler commented Jun 10, 2022 •

edited

Loading

grubberr commented Jun 10, 2022 •

edited by github-actions bot

Loading