Avro destinations: map airbyte_type: integer to long by edgao · Pull Request #25570 · airbytehq/airbyte

Edward Gao (edgao) · 2023-04-26T17:05:27Z

What

#23466 only fixed this behavior for {type: integer}. This PR adds the fix for {type: number, airbyte_type: integer}.

Also updated test cases:

add a test case for {type: integer}
update the avroSchema for {type: number, airbyte_type: integer} to be long

(also, submitted an issue to clean up the data types stuff - it's a pretty messy change so will do it separately https://github.com/airbytehq/airbyte-internal-issues/issues/1726)

github-actions · 2023-04-26T17:10:38Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (10)

Connector	Version	Changelog	Publish
`destination-bigquery`	`1.4.0`	✅	✅
`destination-bigquery-denormalized`	`1.4.0`	✅	✅
`destination-databricks`	`1.0.2`	✅	✅
`destination-gcs`	`0.3.0`	✅	✅
`destination-r2`	`0.1.0`	✅	✅
`destination-redshift`	`0.4.7`	✅	✅
`destination-s3`	`0.4.0`	✅	✅
`destination-s3-glue`	`0.1.7`	✅	✅
`destination-snowflake`	`1.0.1`	✅	✅
`destination-starburst-galaxy`	`0.0.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

Evan Tahler (evantahler)

Tests pass, so !

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com>

…ess (#25588) * add AIRBYTE_ENTRYPOINT env var for kube * amazing, absolute genius * version bump + changelog * derp, no need to publish denormalizeid * fix changelog entry * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Edward Gao (edgao) · 2023-04-28T22:59:17Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/4835432753
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/4835432753
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

Edward Gao (edgao) · 2023-04-28T22:59:21Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/4835433010
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/4835433010
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

Edward Gao (edgao) · 2023-04-28T22:59:24Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/4835433278
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/4835433278
No Python unittests run

Build Passed

Test summary info:

All Passed

Edward Gao (edgao) · 2023-04-28T22:59:27Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/4835433584
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/4835433584
No Python unittests run

Build Passed

Test summary info:

All Passed

Edward Gao (edgao) · 2023-05-05T13:18:54Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4893854973
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4893854973
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

Edward Gao (edgao) · 2023-05-05T15:21:34Z

/publish connector=connectors/destination-bigquery,connectors/destination-bigquery-denormalized,connectors/destination-gcs,connectors/destination-s3,connectors/destination-snowflake

🕑 Publishing the following connectors:
connectors/destination-bigquery,connectors/destination-bigquery-denormalized,connectors/destination-gcs,connectors/destination-s3,connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4894973750

Connector	Version	Did it publish?	Were definitions generated?
connectors/destination-gcs	0.3.0	✅	✅
connectors/destination-bigquery-denormalized	1.4.0	✅	✅
connectors/destination-bigquery	1.4.0	✅	✅
connectors/destination-s3	0.4.0	✅	✅
connectors/destination-snowflake	1.0.1	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Tyler Bernstein (tybernstein) · 2023-05-10T12:57:07Z

Edward Gao (@edgao) is there a chance that this would change the datatype format in the following manner:

1676500610 now formatted as 1.678177401E9?

Edward Gao (edgao) · 2023-05-10T14:45:33Z

I would not expect that - that actually sounds like it's being treated as a number/float. Probably easier to discuss over slack / separate issue though

…#25570) * map the other integer schema to long * fix test + add test * delete_public_access_block for bucket if public (airbytehq#25663) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * 🐛 Source Facebook Marketing: fix `expected records` for CAT (airbytehq#25604) * publish normalization (airbytehq#25591) * publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com> * Bump Airbyte version from 0.44.2 to 0.44.3 * Destination Bigquery: update AIRBYTE_ENTRYPOINT env var for kube process (airbytehq#25588) * add AIRBYTE_ENTRYPOINT env var for kube * amazing, absolute genius * version bump + changelog * derp, no need to publish denormalizeid * fix changelog entry * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * version bumps + changelog * whoops * bump metadata * bump metadatas * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version * auto-bump connector version --------- Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Jeff Cowan (Airbyte) <4992320+jcowanpdx@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: benmoriceau <benmoriceau@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

map the other integer schema to long

c075b5d

This comment was marked as outdated.

Sign in to view

fix test + add test

6b73086

Edward Gao (edgao) marked this pull request as ready for review April 26, 2023 17:43

Edward Gao (edgao) requested a review from a team as a code owner April 26, 2023 17:43

Edward Gao (edgao) changed the title ~~map the other integer schema to long~~ Destinations S3, Bigquery-denormalized: map airbyte_type: integer to long Apr 26, 2023

This comment was marked as outdated.

Sign in to view

Evan Tahler (evantahler) approved these changes Apr 26, 2023

View reviewed changes

Merge branch 'master' into edgao/dest_s3_ints

e4429ae

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into edgao/dest_s3_ints

b5f59cf

This comment was marked as outdated.

Sign in to view

Serhii Chvaliuk (grubberr) and others added 6 commits April 28, 2023 15:54

delete_public_access_block for bucket if public (#25663)

e0ba2d0

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

🐛 Source Facebook Marketing: fix expected records for CAT (#25604)

5f97ebb

publish normalization (#25591)

688c3d5

* publish normalization * bump normalization container version in all the destinations that use it Co-authored-by: Edward Gao <edward.gao@airbyte.io> Co-authored-by: edgao <edgao@users.noreply.github.com>

Bump Airbyte version from 0.44.2 to 0.44.3

86c1d88

version bumps + changelog

81a2259

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/s3 label Apr 28, 2023

Merge branch 'master' into edgao/dest_s3_ints

341b60c

Edward Gao (edgao) changed the title ~~Destinations S3, Bigquery-denormalized: map airbyte_type: integer to long~~ Avro destinations: map airbyte_type: integer to long Apr 28, 2023

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into edgao/dest_s3_ints

47bef9f

This comment was marked as outdated.

Sign in to view

Edward Gao (edgao) added 2 commits May 4, 2023 16:35

whoops

865b153

bump metadata

731d03f

bump metadatas

d085292

Octavia Squidington III (octavia-squidington-iii) added 5 commits May 5, 2023 15:39

auto-bump connector version

536f61a

auto-bump connector version

198b1a1

auto-bump connector version

591eeeb

auto-bump connector version

2de772e

auto-bump connector version

6d8512a

Edward Gao (edgao) enabled auto-merge (squash) May 5, 2023 18:56

Edward Gao (edgao) merged commit ccb6a07 into master May 5, 2023

Edward Gao (edgao) deleted the edgao/dest_s3_ints branch May 5, 2023 19:14

Conversation

Edward Gao (edgao) commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Uh oh!

This comment was marked as outdated.

github-actions Bot commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Affected Connector Report

✅ Sources (0)

✅ Destinations (10)

👀 Other Modules (1)

Actionable Items

Uh oh!

This comment was marked as outdated.

Evan Tahler (evantahler) left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Edward Gao (edgao) commented Apr 28, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Edward Gao (edgao) commented Apr 28, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Edward Gao (edgao) commented Apr 28, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Edward Gao (edgao) commented Apr 28, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Edward Gao (edgao) commented May 5, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Edward Gao (edgao) commented May 5, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tyler Bernstein (tybernstein) commented May 10, 2023

Uh oh!

Edward Gao (edgao) commented May 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Edward Gao (edgao) commented Apr 26, 2023 •

edited

Loading

github-actions Bot commented Apr 26, 2023 •

edited

Loading

Edward Gao (edgao) commented Apr 28, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Apr 28, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Apr 28, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented Apr 28, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented May 5, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented May 5, 2023 •

edited by github-actions Bot

Loading