BigQuery Destination : Fix GCS processing of Facebook data by DoNotPanicUA · Pull Request #9415 · airbytehq/airbyte

Andrii Leonets (DoNotPanicUA) · 2022-01-11T16:48:07Z

What

GCS BigQuery processing uses Avro files.
If field name contains char -, the loader fails due to invalid namesapce value.
Issue OnCall #86

How

Replace - to _ in the field name and use updated names in the namespaces.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

Andrii Leonets (DoNotPanicUA) · 2022-01-11T16:49:25Z

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1683599893
✅ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1683599893
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

Andrii Leonets (DoNotPanicUA) · 2022-01-11T16:49:38Z

/test connector=destination-s3

🕑 destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1683600796
✅ destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1683600796
No Python unittests run

Andrii Leonets (DoNotPanicUA) · 2022-01-11T16:49:47Z

/test connector=destination-gcs

🕑 destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1683601551
✅ destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1683601551
No Python unittests run

Andrii Leonets (DoNotPanicUA) · 2022-01-11T18:14:44Z

/test connector=destination-bigquery-denormalized

🕑 destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1683958140
❌ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1683958140
🐛

The fail is not related to the change. Locally it works fine. The test instance can't reach airbyte/normalization:dev image.

The same test fail on the master: https://github.com/airbytehq/airbyte/actions/runs/1683960600

Andrii Leonets (DoNotPanicUA) · 2022-01-12T11:58:47Z

In addition, integration service has obvious issues. Which might lead to test failure.

Andrii Leonets (DoNotPanicUA) · 2022-01-12T17:19:37Z

/publish connector=connectors/destination-s3

Andrii Leonets (DoNotPanicUA) · 2022-01-12T17:19:51Z

/publish connector=connectors/destination-bigquery

Andrii Leonets (DoNotPanicUA) · 2022-01-12T17:19:59Z

/publish connector=connectors/destination-bigquery-denormalized

# Conflicts: # docs/integrations/destinations/s3.md

Andrii Leonets (DoNotPanicUA) · 2022-01-12T19:57:58Z

/publish connector=connectors/destination-s3

Andrii Leonets (DoNotPanicUA) · 2022-01-12T19:58:14Z

/publish connector=connectors/destination-bigquery

Andrii Leonets (DoNotPanicUA) · 2022-01-12T19:58:23Z

/publish connector=connectors/destination-bigquery-denormalized

Andrii Leonets (DoNotPanicUA) · 2022-01-13T13:13:07Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1692853484
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1692853484

Andrii Leonets (DoNotPanicUA) · 2022-01-13T13:13:16Z

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1692854408
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1692854408

Andrii Leonets (DoNotPanicUA) · 2022-01-13T13:13:23Z

/publish connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1692854866
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/1692854866

Andrii Leonets (DoNotPanicUA) · 2022-01-13T14:59:49Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693321202
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693321202

Andrii Leonets (DoNotPanicUA) · 2022-01-13T16:00:30Z

/publish connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693612171
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1693612171

Andrii Leonets (DoNotPanicUA) added 2 commits January 11, 2022 18:43

Fix GCS Avro file processing with invalid "-" character

6b343c6

Extend test data to cover the case

21641ba

Andrii Leonets (DoNotPanicUA) self-assigned this Jan 11, 2022

Andrii Leonets (DoNotPanicUA) requested review from Oleksandr Sheheda (alexandr-shegeda) and Liren Tu (tuliren) January 11, 2022 16:48

github-actions Bot added the area/connectors Connector related issues label Jan 11, 2022

This comment has been minimized.

Sign in to view

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 11, 2022 16:51 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 11, 2022 16:52 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 11, 2022 17:20 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 11, 2022 18:17 Inactive

Liren Tu (tuliren) approved these changes Jan 12, 2022

View reviewed changes

Oleksandr Sheheda (alexandr-shegeda) approved these changes Jan 12, 2022

View reviewed changes

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 12, 2022 16:04 Inactive

incr ver

d3ea38e

github-actions Bot added the area/documentation Improvements or additions to documentation label Jan 12, 2022

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 12, 2022 17:10 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 12, 2022 17:21 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 12, 2022 17:22 Inactive

Andrii Leonets (DoNotPanicUA) added 2 commits January 12, 2022 21:55

Merge remote-tracking branch 'origin/master' into aleonets/oncall-86

18655ef

# Conflicts: # docs/integrations/destinations/s3.md

s3 ver upd

1245daf

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 12, 2022 19:58 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 12, 2022 20:00 Inactive

Merge remote-tracking branch 'origin/master' into aleonets/oncall-86

c6e6b88

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 13, 2022 13:14 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 13, 2022 13:15 Inactive

add dependency

ee5c236

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 13, 2022 15:00 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 13, 2022 15:02 Inactive

add dependency

121e6a7

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 13, 2022 15:57 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 13, 2022 16:02 Inactive

Andrii Leonets (DoNotPanicUA) temporarily deployed to more-secrets January 13, 2022 16:49 Inactive

Andrii Leonets (DoNotPanicUA) merged commit c904b44 into master Jan 13, 2022

Andrii Leonets (DoNotPanicUA) deleted the aleonets/oncall-86 branch January 13, 2022 17:16

Octavia Squidington III (octavia-squidington-iii) mentioned this pull request Jan 20, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9630

Merged

Uh oh!

Conversation

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 12, 2022

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 11, 2022 •

edited

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 •

edited by github-actions Bot

Loading

Andrii Leonets (DoNotPanicUA) commented Jan 13, 2022 •

edited by github-actions Bot

Loading