Enable full SAT for the BigQuery source #19720

DoNotPanicUA · 2022-11-22T16:46:57Z

What

Enable SAT for the BigQuery source

Added secrets:

SECRET_SOURCE-BIGQUERY_SAT__CREDS

Configured test instances

[BigQuery] sat_test_dataset

Found issues

SAT related issues:

SAT can't check null values due to column type validation fail

Source related issues:

[BigQuery] Complex type like REPEATED or OBJECT fail validation. The root cause of the issue requires investigation. Is it SAT issue, or is the source work wrong?
[BigQuery] Fails the future incremental check. An investigation is required. Or the abnormal cursor value format is wrong, or the source doesn't accept the initial state message.

DoNotPanicUA · 2022-12-01T12:35:29Z

Yes!

I'm curious how the BQ instance for testing was setup. Other than in the Google Secrets, should we document the table/database/user anywhere?

It would be very useful to store all links to the credentials and creation of test instances.
So far, I store all scripts in the repository like here.
We could create a doc as they don't contain any sensitive information.

evantahler · 2022-12-01T21:08:36Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3596550597
❌ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3596550597
🐛

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
============ 29 passed, 1 skipped, 31 warnings in 61.86s (0:01:01) =============

evantahler

🥳

alafanechere

If you commit the SQL files I would suggest documenting how you provisioned BigQuery with the test data. It's strange that the /test reported status is "failing" while the tests have passed. I'm going to re-run it to check if it's a transient problem.

airbyte-integrations/connectors/source-bigquery/integration_tests/basic_dataset.sql

airbyte-integrations/connectors/source-bigquery/acceptance-test-config.yml

alafanechere · 2022-12-02T12:37:36Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3601749168
❌ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3601749168
🐛

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
================= 29 passed, 1 skipped, 31 warnings in 57.88s ==================

DoNotPanicUA · 2022-12-02T12:48:11Z

alafanechere · 2022-12-02T14:42:45Z

@DoNotPanicUA would you mind trying to update your branch? I'm wondering if the Report status failure is due to recent changes on the AMI (cc @evantahler )

DoNotPanicUA · 2022-12-02T15:59:48Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3603180470
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3603180470
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
============ 29 passed, 1 skipped, 31 warnings in 60.49s (0:01:00) =============

evantahler · 2022-12-02T16:42:23Z

@DoNotPanicUA would you mind trying to update your branch? I'm wondering if the Report status failure is due to recent changes on the AMI (cc @evantahler )

Yep! The AMI was fixed yesterday here #19949 for /test and /publish commands. Looking good now!

alafanechere

👍 Please create a script or a README under integration_tests to explain how to run the sql queries you committed.

airbyte-integrations/connectors/source-bigquery/acceptance-test-config.yml

DoNotPanicUA · 2022-12-07T14:42:00Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3639933799
❌ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3639933799
🐛

alafanechere

Requesting changes for the reasons explained here: #19915 (comment)

alafanechere · 2022-12-07T21:03:00Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3642803432
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3642803432
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
======================== 29 passed, 1 skipped in 59.41s ========================

alafanechere

I updated your branch to get the test working again. Feel free to merge once they pass.

DoNotPanicUA · 2022-12-08T11:17:42Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3647778836
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/3647778836
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
======================== 29 passed, 1 skipped in 56.75s ========================

enable full SAT for the BigQuery source

7cd9b3f

DoNotPanicUA linked an issue Nov 22, 2022 that may be closed by this pull request

[SAT] [JAVA] Enable DISCOVER and READ SAT for Java sources with global instances #19708

Closed

octavia-squidington-iv added area/connectors Connector related issues connectors/source/bigquery labels Nov 22, 2022

This comment was marked as outdated.

Sign in to view

DoNotPanicUA added 2 commits November 28, 2022 13:12

use basic database instead of full due to SAT fails

92184c2

add full dataset sql

ce727c2

This comment was marked as outdated.

Sign in to view

DoNotPanicUA self-assigned this Nov 28, 2022

minor changes

e8d7254

This comment was marked as outdated.

Sign in to view

octavia-squidington-iv added the connectors/source/redshift label Nov 29, 2022

DoNotPanicUA changed the title ~~enable full SAT for the BigQuery source~~ [SAT] Enable full SAT for the global sources Nov 29, 2022

This comment was marked as off-topic.

Sign in to view

pedroslopez removed a link to an issue Nov 29, 2022

[SAT] [JAVA] Enable DISCOVER and READ SAT for Java sources with global instances #19708

Closed

This was linked to issues Nov 29, 2022

Enable SAT for hosted sources - BigQuery #19894

Closed

Enable SAT for hosted sources - Redshift #19895

Closed

DoNotPanicUA changed the title ~~[SAT] Enable full SAT for the global sources~~ [SAT] Enable full SAT for the BigQuery source Nov 30, 2022

DoNotPanicUA changed the title ~~[SAT] Enable full SAT for the BigQuery source~~ Enable full SAT for the BigQuery source Nov 30, 2022

DoNotPanicUA force-pushed the aleonets/19708-enable-full-sat-host-src branch from 95ded5f to e8d7254 Compare November 30, 2022 11:58

Merge branch 'master' into aleonets/19708-enable-full-sat-host-src

023f308

This comment was marked as outdated.

Sign in to view

DoNotPanicUA removed a link to an issue Nov 30, 2022

Enable SAT for hosted sources - Redshift #19895

Closed

This comment was marked as outdated.

Sign in to view

Merge branch 'master' into aleonets/19708-enable-full-sat-host-src

d1a42f7

DoNotPanicUA marked this pull request as ready for review November 30, 2022 15:53

DoNotPanicUA requested a review from a team as a code owner November 30, 2022 15:53

DoNotPanicUA requested review from evantahler and a team November 30, 2022 15:53

DoNotPanicUA requested a review from evantahler December 1, 2022 12:35

evantahler requested a review from alafanechere December 1, 2022 21:07

evantahler approved these changes Dec 1, 2022

View reviewed changes

alafanechere requested changes Dec 2, 2022

View reviewed changes

airbyte-integrations/connectors/source-bigquery/integration_tests/basic_dataset.sql Show resolved Hide resolved

airbyte-integrations/connectors/source-bigquery/acceptance-test-config.yml Outdated Show resolved Hide resolved

DoNotPanicUA requested a review from alafanechere December 2, 2022 13:00

new line

b64dc0c

Merge branch 'master' into aleonets/19708-enable-full-sat-host-src

cbfa6cf

alafanechere approved these changes Dec 6, 2022

View reviewed changes

DoNotPanicUA mentioned this pull request Dec 6, 2022

Enable full SAT for the Redshift source #19915

Merged

alafanechere requested changes Dec 6, 2022

View reviewed changes

airbyte-integrations/connectors/source-bigquery/acceptance-test-config.yml Show resolved Hide resolved

readme + format

975b478

DoNotPanicUA requested a review from alafanechere December 7, 2022 14:42

alafanechere requested changes Dec 7, 2022

View reviewed changes

Merge branch 'master' into aleonets/19708-enable-full-sat-host-src

984aaa2

alafanechere approved these changes Dec 7, 2022

View reviewed changes

Merge branch 'master' into aleonets/19708-enable-full-sat-host-src

aeadbb5

DoNotPanicUA merged commit 36d4c92 into master Dec 8, 2022

DoNotPanicUA deleted the aleonets/19708-enable-full-sat-host-src branch December 8, 2022 11:55

github-actions bot mentioned this pull request Dec 8, 2022

Bump helm chart version reference to 0.43.0 #20268

Closed

octavia-squidington-iii mentioned this pull request Dec 9, 2022

Bump Airbyte version from 0.40.24 to 0.40.25 #20279

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable full SAT for the BigQuery source #19720

Enable full SAT for the BigQuery source #19720

DoNotPanicUA commented Nov 22, 2022 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as outdated.

DoNotPanicUA commented Dec 1, 2022

evantahler commented Dec 1, 2022 •

edited by github-actions bot

Loading

evantahler left a comment

alafanechere left a comment

alafanechere commented Dec 2, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Dec 2, 2022

alafanechere commented Dec 2, 2022

DoNotPanicUA commented Dec 2, 2022 •

edited by github-actions bot

Loading

evantahler commented Dec 2, 2022 •

edited

Loading

alafanechere left a comment

DoNotPanicUA commented Dec 7, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

alafanechere commented Dec 7, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

DoNotPanicUA commented Dec 8, 2022 •

edited by github-actions bot

Loading

Enable full SAT for the BigQuery source #19720

Enable full SAT for the BigQuery source #19720

Conversation

DoNotPanicUA commented Nov 22, 2022 • edited Loading

What

Added secrets:

Configured test instances

Found issues

SAT related issues:

Source related issues:

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as outdated.

DoNotPanicUA commented Dec 1, 2022

evantahler commented Dec 1, 2022 • edited by github-actions bot Loading

Build Passed

evantahler left a comment

Choose a reason for hiding this comment

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere commented Dec 2, 2022 • edited by github-actions bot Loading

Build Passed

DoNotPanicUA commented Dec 2, 2022

alafanechere commented Dec 2, 2022

DoNotPanicUA commented Dec 2, 2022 • edited by github-actions bot Loading

Build Passed

evantahler commented Dec 2, 2022 • edited Loading

alafanechere left a comment

Choose a reason for hiding this comment

DoNotPanicUA commented Dec 7, 2022 • edited by github-actions bot Loading

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere commented Dec 7, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere left a comment

Choose a reason for hiding this comment

DoNotPanicUA commented Dec 8, 2022 • edited by github-actions bot Loading

Build Passed

DoNotPanicUA commented Nov 22, 2022 •

edited

Loading

evantahler commented Dec 1, 2022 •

edited by github-actions bot

Loading

alafanechere commented Dec 2, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Dec 2, 2022 •

edited by github-actions bot

Loading

evantahler commented Dec 2, 2022 •

edited

Loading

DoNotPanicUA commented Dec 7, 2022 •

edited by github-actions bot

Loading

alafanechere commented Dec 7, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Dec 8, 2022 •

edited by github-actions bot

Loading