🐛 Fix Snowflake destination normalization to accept any date-time format. #6052

yaroslav-dudar · 2021-09-14T10:02:41Z

What

Fixed snowflake destination support for date-time format #5713

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…udar/5713

yaroslav-dudar · 2021-09-14T11:51:30Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1233489470
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1233489470
🐛 https://gradle.com/s/yrcxnlhnh2lli

sherifnada · 2021-09-15T02:58:34Z

@yaroslav-dudar your build issue should be fixed once you merge latest master

…udar/5713

yaroslav-dudar · 2021-09-15T06:22:38Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1236498147
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1236498147
🐛

airbyte-integrations/bases/base-normalization/Dockerfile

yaroslav-dudar · 2021-09-15T13:41:57Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1237854943
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1237854943
🐛 https://gradle.com/s/nd5narrjd7oxo

yaroslav-dudar · 2021-09-15T14:31:13Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1238034385
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1238034385
🐛 https://gradle.com/s/w4svts4asovue

yaroslav-dudar · 2021-09-15T17:49:20Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1238675382
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1238675382

yaroslav-dudar · 2021-09-21T18:21:04Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1258802649
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1258802649

sherifnada · 2021-09-22T05:31:24Z

@yaroslav-dudar I think the failure is in the test itself. Have you verified they are passing locally?

yaroslav-dudar · 2021-09-22T06:14:52Z

@yaroslav-dudar I think the failure is in the test itself. Have you verified they are passing locally?

@sherifnada but it's passed here #6052 (comment)

and I see errors like this

docker: Error response from daemon: manifest for airbyte/destination-redshift:dev not found: manifest unknown: manifest unknown.
docker: Error response from daemon: manifest for airbyte/destination-postgres:dev not found: manifest unknown: manifest unknown.
docker: Error response from daemon: manifest for airbyte/destination-bigquery:dev not found: manifest unknown: manifest unknown.
docker: Error response from daemon: manifest for airbyte/destination-snowflake:dev not found: manifest unknown: manifest unknown.

yaroslav-dudar · 2021-09-22T07:23:08Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1260788282
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1260788282
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         138     39    72%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1034    394    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        45     45     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  776    430    45%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     117      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349    199    43%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         138     39    72%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1034    394    62%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    11      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     117      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 10      0   100%
	 normalization/transform_catalog/stream_processor.py                 349     32    91%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         138     45    67%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1034    180    83%

yaroslav-dudar · 2021-09-22T07:56:27Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1260883045
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1260883045

ChristopheDuong · 2021-09-22T10:01:19Z

Yes, if you see these errors:

docker: Error response from daemon: manifest for airbyte/destination-redshift:dev not found: manifest unknown: manifest unknown.
docker: Error response from daemon: manifest for airbyte/destination-postgres:dev not found: manifest unknown: manifest unknown.
docker: Error response from daemon: manifest for airbyte/destination-bigquery:dev not found: manifest unknown: manifest unknown.
docker: Error response from daemon: manifest for airbyte/destination-snowflake:dev not found: manifest unknown: manifest unknown.

Then, for some reason, when you publish, maybe the destination image (dev versions) are not available when publishing but they are correctly built/used during the test?

yaroslav-dudar · 2021-09-22T10:06:56Z

it's weird because /test is ok, but /publish failed with Unable to find image 'airbyte/destination-postgres:dev' locally, Unable to find image 'airbyte/destination-bigquery:dev' locally ...

yaroslav-dudar · 2021-09-22T12:33:31Z

@ChristopheDuong
I see that Task :airbyte-integrations:bases:base-normalization:customIntegrationTests comes before
:airbyte-integrations:connectors:destination-snowflake:airbyteDocker , :airbyte-integrations:connectors:destination-mysql:airbyteDocker etc . I think this may be a problem as we run tests before building destination images

https://github.com/airbytehq/airbyte/runs/3672622870?check_suite_focus=true#step:10:4645 - base-normalization:customIntegrationTests
https://github.com/airbytehq/airbyte/runs/3672622870?check_suite_focus=true#step:10:4772 - :airbyte-integrations:connectors:destination-snowflake:airbyteDocker

yaroslav-dudar · 2021-09-22T13:40:50Z

maybe I need to change integrationTest.dependsOn("customIntegrationTestPython") to customIntegrationTests.dependsOn("customIntegrationTestPython") here https://github.com/airbytehq/airbyte/blob/yaroslav-dudar/5713/airbyte-integrations/bases/base-normalization/build.gradle#L50?
@sherifnada @ChristopheDuong ?

ChristopheDuong · 2021-09-22T14:27:18Z

Oh, I remember I did struggle the same as you in the past...

here, see this PR: https://github.com/airbytehq/airbyte/pull/4910/files#r677136914
I fixed the CI for /test commands in that PR but not for /publish so it explains why you are having difficulty with the publish...

yaroslav-dudar · 2021-09-22T16:01:51Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1262549835
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1262549835

yaroslav-dudar · 2021-09-22T17:02:39Z

@ChristopheDuong please check, is it ok for you 5b3bd3a ?

another point, seems like connector was published but I got error in the end https://github.com/airbytehq/airbyte/runs/3677499341?check_suite_focus=true#step:10:12179

is it possible that my changes are the reason of that issue??

ChristopheDuong · 2021-09-22T17:07:25Z

@ChristopheDuong please check, is it ok for you 5b3bd3a ?

another point, seems like connector was published but I got error in the end https://github.com/airbytehq/airbyte/runs/3677499341?check_suite_focus=true#step:10:12179

is it possible that my changes are the reason of that issue??

Yes, it seemed to have publish fine!

Can you check that published docker image version has your change in it? (not in conflict from a publish from another PR?)

ChristopheDuong · 2021-09-22T17:10:31Z

The error seemed to come from:

docker run --rm airbyte/normalization:0.1.45 spec
+ jq -R 'fromjson? | .'
+ jq -s 'map(select(.spec != null)) | map(.spec) | first | if . != null then . else error("no spec found") end'
Unknown command: spec
jq: error (at <stdin>:0): no spec found

which is from #5994

Normalization does not have "spec" commands as usual connectors do... so it seems normal and the publish should probably ignore writing to spec cache for normalization images in tools/integrations/manage.sh, right @cgardens?

yaroslav-dudar · 2021-09-22T17:37:22Z

@ChristopheDuong please check, is it ok for you 5b3bd3a ?
another point, seems like connector was published but I got error in the end https://github.com/airbytehq/airbyte/runs/3677499341?check_suite_focus=true#step:10:12179
is it possible that my changes are the reason of that issue??

Yes, it seemed to have publish fine!

Can you check that published docker image version has your change in it? (not in conflict from a publish from another PR?)

verified, airbyte/normalization:0.1.45 includes my changes

yaroslav-dudar · 2021-09-23T06:19:23Z

@ChristopheDuong please notify me if I can merge it

ChristopheDuong · 2021-09-23T07:54:04Z

Yes, if it's published, you're all good to merge, go ahead

yaroslav-dudar added 4 commits September 13, 2021 20:44

snowflake date-time parsing

325579e

upd date-time snowflake parser

a5d63c9

Merge branch 'master' of github.com:airbytehq/airbyte into yaroslav-d…

6f961bf

…udar/5713

bump version

38422a9

github-actions bot added the normalization label Sep 14, 2021

yaroslav-dudar changed the title ~~Fix Snowflake destination normalization to accept any date-time format.~~ 🐛 Fix Snowflake destination normalization to accept any date-time format. Sep 14, 2021

jrhizor temporarily deployed to more-secrets September 14, 2021 10:06 Inactive

yaroslav-dudar linked an issue Sep 14, 2021 that may be closed by this pull request

Fix Snowflake destination normalization to accept any date-time format. #5713

Closed

jrhizor temporarily deployed to more-secrets September 14, 2021 10:30 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into yaroslav-d…

5bdb41c

…udar/5713

jrhizor temporarily deployed to more-secrets September 14, 2021 11:53 Inactive

yaroslav-dudar requested review from bazarnov and midavadim September 14, 2021 13:45

Merge branch 'master' of github.com:airbytehq/airbyte into yaroslav-d…

78f19b5

…udar/5713

jrhizor temporarily deployed to more-secrets September 15, 2021 06:25 Inactive

midavadim reviewed Sep 15, 2021

View reviewed changes

airbyte-integrations/bases/base-normalization/Dockerfile Show resolved Hide resolved

fix bigquery creds

b0dd2b2

jrhizor temporarily deployed to more-secrets September 15, 2021 13:44 Inactive

fix bigquery creds parsing

93f46f6

jrhizor temporarily deployed to more-secrets September 15, 2021 14:33 Inactive

fix mysql failed tests

ac62245

github-actions bot added area/documentation Improvements or additions to documentation area/worker Related to worker labels Sep 15, 2021

jrhizor temporarily deployed to more-secrets September 15, 2021 17:52 Inactive

jrhizor temporarily deployed to more-secrets September 21, 2021 18:23 Inactive

jrhizor temporarily deployed to more-secrets September 22, 2021 07:25 Inactive

jrhizor temporarily deployed to more-secrets September 22, 2021 07:58 Inactive

try to fix publish command

5b3bd3a

yaroslav-dudar temporarily deployed to more-secrets September 22, 2021 16:02 Inactive

jrhizor temporarily deployed to more-secrets September 22, 2021 16:04 Inactive

ChristopheDuong mentioned this pull request Sep 22, 2021

publish spec as part of connector publish script #5994

Merged

1 task

yaroslav-dudar merged commit a6ecfda into master Sep 23, 2021

yaroslav-dudar deleted the yaroslav-dudar/5713 branch September 23, 2021 08:10

Phlair mentioned this pull request Sep 23, 2021

Fix ssh tunneling for normalization #6396

Merged

jrhizor mentioned this pull request Sep 25, 2021

Bump Airbyte version from 0.29.21-alpha to 0.29.22-alpha #6450

Merged

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Fix Snowflake destination normalization to accept any date-time format. #6052

🐛 Fix Snowflake destination normalization to accept any date-time format. #6052

yaroslav-dudar commented Sep 14, 2021

yaroslav-dudar commented Sep 14, 2021 •

edited by github-actions bot

Loading

sherifnada commented Sep 15, 2021

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 21, 2021 •

edited by github-actions bot

Loading

sherifnada commented Sep 22, 2021

yaroslav-dudar commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Sep 22, 2021

yaroslav-dudar commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021

ChristopheDuong commented Sep 22, 2021

yaroslav-dudar commented Sep 22, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 22, 2021

ChristopheDuong commented Sep 22, 2021 •

edited

Loading

ChristopheDuong commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021

yaroslav-dudar commented Sep 23, 2021

ChristopheDuong commented Sep 23, 2021

🐛 Fix Snowflake destination normalization to accept any date-time format. #6052

🐛 Fix Snowflake destination normalization to accept any date-time format. #6052

Conversation

yaroslav-dudar commented Sep 14, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

yaroslav-dudar commented Sep 14, 2021 • edited by github-actions bot Loading

sherifnada commented Sep 15, 2021

yaroslav-dudar commented Sep 15, 2021 • edited by github-actions bot Loading

yaroslav-dudar commented Sep 15, 2021 • edited by github-actions bot Loading

yaroslav-dudar commented Sep 15, 2021 • edited by github-actions bot Loading

yaroslav-dudar commented Sep 15, 2021 • edited by github-actions bot Loading

yaroslav-dudar commented Sep 21, 2021 • edited by github-actions bot Loading

sherifnada commented Sep 22, 2021

yaroslav-dudar commented Sep 22, 2021 • edited Loading

yaroslav-dudar commented Sep 22, 2021 • edited by github-actions bot Loading

yaroslav-dudar commented Sep 22, 2021 • edited by github-actions bot Loading

ChristopheDuong commented Sep 22, 2021

yaroslav-dudar commented Sep 22, 2021 • edited Loading

yaroslav-dudar commented Sep 22, 2021 • edited Loading

yaroslav-dudar commented Sep 22, 2021

ChristopheDuong commented Sep 22, 2021

yaroslav-dudar commented Sep 22, 2021 • edited by github-actions bot Loading

yaroslav-dudar commented Sep 22, 2021

ChristopheDuong commented Sep 22, 2021 • edited Loading

ChristopheDuong commented Sep 22, 2021 • edited Loading

yaroslav-dudar commented Sep 22, 2021

yaroslav-dudar commented Sep 23, 2021

ChristopheDuong commented Sep 23, 2021

yaroslav-dudar commented Sep 14, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 15, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 21, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited by github-actions bot

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited

Loading

yaroslav-dudar commented Sep 22, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Sep 22, 2021 •

edited

Loading

ChristopheDuong commented Sep 22, 2021 •

edited

Loading