🐛 Destination BigQuery (+denormalized): correctly parse buffer count by edgao · Pull Request #26213 · airbytehq/airbyte

Edward Gao (edgao) · 2023-05-17T22:42:45Z

we're not actually parsing the config correctly (it's nested under the loading_method object). Extract it correctly.

also swap the order of parameters in the assertEquals calls. The convention is assertEquals(expected, actual).

github-actions · 2023-05-17T22:43:05Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
You, or an Airbyter, have run /publish successfully on this PR - or on a non-forked branch
You've updated the connector's metadata.yaml file new!

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-05-17T22:47:33Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (2)

Connector	Version	Changelog	Publish
`destination-bigquery`	`1.4.3`	✅	❌ (diff seed version)
`destination-bigquery-denormalized`	`1.4.1`	✅	❌ (diff seed version)

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

Edward Gao (edgao) · 2023-05-17T22:56:27Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/5008629473
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/5008629473
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

Edward Gao (edgao) · 2023-05-17T22:56:31Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/5008629916
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/5008629916
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

Ryan Fu (ryankfu)

LGTM, thanks for getting this up!

Edward Gao (edgao) · 2023-05-17T23:49:42Z

/publish connector=connectors/destination-bigquery,connectors/destination-bigquery-denormalized

🕑 Publishing the following connectors:
connectors/destination-bigquery,connectors/destination-bigquery-denormalized
https://github.com/airbytehq/airbyte/actions/runs/5008946057

| Connector | Version | Did it publish? |
| --- | --- | --- | --- |
| connectors/destination-bigquery-denormalized | 1.4.1 | ✅ |
| connectors/destination-bigquery | 1.4.2 | ❌ |

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…25925) * readme update * allow passing additional flags to test containers * remove build dependency * Automated Change * versioning updates * restore denormalized change from master * formatting changes * formatting * Automated Change * update metadata file --------- Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>

* Source Airtable: skip missing streams * Move stream removal to a separate method, cover with tests * Update changelog * Fix flake warnings * Update docs/integrations/sources/airtable.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Update docs/integrations/sources/airtable.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Automated Change * Update link to docs in warning * Automated Change * Automated Change * Automated Change * “Empty-Commit” --------- Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: arsenlosenko <arsenlosenko@users.noreply.github.com>

* Initial commit - All test passed * add stream fax cover * refactor docs * fix schema, Added pagination * Add several streams, fix schema * fix schema, add streams, refactor docs * EOF * Resolve conflicts * Resolve conflicts * add metadata file --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>

Edward Gao (edgao) · 2023-05-18T14:38:42Z

/publish connector=connectors/destination-bigquery

🕑 Publishing the following connectors:
connectors/destination-bigquery
https://github.com/airbytehq/airbyte/actions/runs/5015106490

| Connector | Version | Did it publish? |
| --- | --- | --- | --- |
| connectors/destination-bigquery | 1.4.3 | ✅ |

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

table is currently broken, e.g. #26213 (comment)

…from config (airbytehq#26213) * fix logic in parsing config * simplify logic * ugh * holy moly that took way too many iterations * version bumps / changelog * Automated Change * ✨ Destination Bigquery: stop running normalization container for DAT (airbytehq#25925) * readme update * allow passing additional flags to test containers * remove build dependency * Automated Change * versioning updates * restore denormalized change from master * formatting changes * formatting * Automated Change * update metadata file --------- Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> * fix version (airbytehq#26218) * Source Airtable: skip missing streams (airbytehq#25946) * Source Airtable: skip missing streams * Move stream removal to a separate method, cover with tests * Update changelog * Fix flake warnings * Update docs/integrations/sources/airtable.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Update docs/integrations/sources/airtable.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * Automated Change * Update link to docs in warning * Automated Change * Automated Change * Automated Change * “Empty-Commit” --------- Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: arsenlosenko <arsenlosenko@users.noreply.github.com> * 🎉 New Source: Ringcentral [Low code CDK] (airbytehq#25701) * Initial commit - All test passed * add stream fax cover * refactor docs * fix schema, Added pagination * Add several streams, fix schema * fix schema, add streams, refactor docs * EOF * Resolve conflicts * Resolve conflicts * add metadata file --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * rebump version * Automated Change --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: arsenlosenko <arsenlosenko@users.noreply.github.com> Co-authored-by: btkcodedev <btk.codedev@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>

Edward Gao (edgao) added 2 commits May 17, 2023 15:39

fix logic in parsing config

e63a237

simplify logic

f526de6

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/destination/bigquery connectors/destination/bigquery-denormalized labels May 17, 2023

This comment was marked as outdated.

Sign in to view

ugh

f1e7236

This comment was marked as outdated.

Sign in to view

holy moly that took way too many iterations

c4b4c0d

Edward Gao (edgao) marked this pull request as ready for review May 17, 2023 22:59

Edward Gao (edgao) requested a review from a team as a code owner May 17, 2023 22:59

Edward Gao (edgao) changed the title ~~Destination BigQuery (+denormalized): correctly parse buffer count~~ 🐛 Destination BigQuery (+denormalized): correctly parse buffer count May 17, 2023

Ryan Fu (ryankfu) approved these changes May 17, 2023

View reviewed changes

version bumps / changelog

ed22d32

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label May 17, 2023

Automated Change

65d5e1c

Joe Bell (jbfbell) and others added 5 commits May 18, 2023 07:34

fix version (#26218)

d2bcf84

rebump version

ff0a3d8

Octavia Squidington III (octavia-squidington-iii) added normalization connectors/source/airtable labels May 18, 2023

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/ringcentral label May 18, 2023

Merge branch 'master' into edgao/dest_bq_parse_file_buffer_count_config

30e913d

Octavia Squidington III (octavia-squidington-iii) removed the normalization label May 18, 2023

Edward Gao (edgao) added a commit that referenced this pull request May 18, 2023

fix publish command output

6259caa

table is currently broken, e.g. #26213 (comment)

Edward Gao (edgao) mentioned this pull request May 18, 2023

tiny fix in /publish output #26267

Merged

Automated Change

95ddac1

Edward Gao (edgao) enabled auto-merge (squash) May 18, 2023 15:25

Merge branch 'master' into edgao/dest_bq_parse_file_buffer_count_config

6d9fb6b

Edward Gao (edgao) added the checklist-action-run label May 18, 2023

Edward Gao (edgao) and others added 3 commits May 18, 2023 08:36

Merge branch 'master' into edgao/dest_bq_parse_file_buffer_count_config

1f99688

Automated Change

a1f49ac

dumb change to trigger ci

8cfe989

Edward Gao (edgao) merged commit c25afc4 into master May 18, 2023

Edward Gao (edgao) deleted the edgao/dest_bq_parse_file_buffer_count_config branch May 18, 2023 16:45

glevineLeap mentioned this pull request May 19, 2023

Source Postgres: Initial Sync for CDC Connection Fails due to <100% of Records Committed #26270

Closed

1 task

Navigation Menu

Repository navigation

🐛 Destination BigQuery (+denormalized): correctly parse buffer count - #26213

🐛 Destination BigQuery (+denormalized): correctly parse buffer count#26213
Edward Gao (edgao) merged 17 commits into
masterfrom
edgao/dest_bq_parse_file_buffer_count_config

Edward Gao (edgao) commented May 17, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented May 17, 2023 •

edited by edgao

Loading

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented May 17, 2023 •

edited

Loading

✅ Sources (0)

❌ Destinations (2)

👀 Other Modules (1)

Actionable Items

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Edward Gao (edgao) commented May 17, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Edward Gao (edgao) commented May 17, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Ryan Fu (ryankfu) left a comment

Uh oh!

Edward Gao (edgao) commented May 17, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Edward Gao (edgao) commented May 18, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

Edward Gao (edgao) commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 17, 2023 • edited by edgao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

github-actions Bot commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Affected Connector Report

✅ Sources (0)

❌ Destinations (2)

👀 Other Modules (1)

Actionable Items

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Edward Gao (edgao) commented May 17, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Edward Gao (edgao) commented May 17, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Ryan Fu (ryankfu) left a comment

Choose a reason for hiding this comment

Uh oh!

Edward Gao (edgao) commented May 17, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Edward Gao (edgao) commented May 18, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Edward Gao (edgao) commented May 17, 2023 •

edited

Loading

github-actions Bot commented May 17, 2023 •

edited by edgao

Loading

github-actions Bot commented May 17, 2023 •

edited

Loading

Edward Gao (edgao) commented May 17, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented May 17, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented May 17, 2023 •

edited by github-actions Bot

Loading

Edward Gao (edgao) commented May 18, 2023 •

edited by github-actions Bot

Loading