🎉 New Source: Big Query #4457

DoNotPanicUA · 2021-07-01T10:24:40Z

What

New source Big Query.
Implemented Full and Incremental refresh types.

Bonus 🍬
Method for generation comprehensive test table SourceComprehensiveTest.getMarkdownTestTable()

Comprehensive tests

Data Type	Insert values	Expected values	Common test result
int64	`null`, `-128`, `127`, `9223372036854775807`, `-9223372036854775808`	`null`, `-128`, `127`, `9223372036854775807`, `-9223372036854775808`	Ok
int	`null`, `-128`, `127`	`null`, `-128`, `127`	Ok
smallint	`null`, `-128`, `127`	`null`, `-128`, `127`	Ok
integer	`null`, `-128`, `127`	`null`, `-128`, `127`	Ok
bigint	`null`, `-128`, `127`	`null`, `-128`, `127`	Ok
tinyint	`null`, `-128`, `127`	`null`, `-128`, `127`	Ok
byteint	`null`, `-128`, `127`	`null`, `-128`, `127`	Ok
numeric	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	Ok
bignumeric	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	Ok
decimal	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	Ok
bigdecimal	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	`null`, `-128`, `127`, `999999999999999999`, `-999999999999999999`, `0.123456789`, `-0.123456789`	Ok
float64	`null`, `-128`, `127`, `0.123456789`, `-0.123456789`	`null`, `-128.0`, `127.0`, `0.123456789`, `-0.123456789`	Ok
bool	`true`, `false`, `null`	`true`, `false`, `null`	Ok
bytes	`FROM_BASE64("test")`, `null`	`test`, `null`	Ok
date	`date('2021-10-20')`, `date('9999-12-31')`, `date('0001-01-01')`, `null`	`2021-10-20T00:00:00Z`, `9999-12-31T00:00:00Z`, `0001-01-01T00:00:00Z`, `null`	Ok
datetime	`datetime('2021-10-20 11:22:33')`, `datetime('9999-12-31 11:22:33')`, `datetime('0001-01-01 11:22:33')`, `null`	`2021-10-20T11:22:33Z`, `9999-12-31T11:22:33Z`, `0001-01-01T11:22:33Z`, `null`	Ok
timestamp	`timestamp('2021-10-20 11:22:33')`, `null`	`2021-10-20T11:22:33Z`, `null`	Ok
geography	`ST_GEOGFROMTEXT('POINT(1 2)')`, `null`	`POINT(1 2)`, `null`	Ok
string	`'qwe'`, `'йцу'`, `null`	`qwe`, `йцу`, `null`	Ok
struct	`STRUCT("B.A",12)`, `null`		Ok
time	`TIME(15, 30, 00)`, `null`	`15:30:00`, `null`	Ok
array	`['a', 'b']`	`[{"test_column":"a"},{"test_column":"b"}]`	Ok
struct	`STRUCT('s' as frst, 1 as sec, STRUCT(555 as id_col, STRUCT(TIME(15, 30, 00) as time) as mega_obbj) as obbj)`	`{"frst":"s","sec":1,"obbj":{"id_col":555,"mega_obbj":{"last_col":"15:30:00"}}}`	Ok

Pre-merge Checklist

…ource)

s not an abstract class. + add default implementation for `AbstractRelationalDbSource.getFullyQualifiedTableName`

…nto aleonets/1876-source-bigquery

…act-source # Conflicts: # airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

…dardSQLTypeName to JsonSchemaPrimitive

# Conflicts: # airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

…nto aleonets/1876-source-bigquery

yaroslav-hrytsaienko

@DoNotPanicUA @heade nice guys, very good job!

yaroslav-hrytsaienko · 2021-07-15T14:50:45Z

...-test/src/main/java/io/airbyte/integrations/standardtest/source/SourceComprehensiveTest.java

+   *
+   * @return formatted list of test cases
+   */
+  public String getMarkdownTestTable() {


@DoNotPanicUA nice!

You can write return the value right here :-)

this is great! next step will be write this into a markdown file as part of the build so it's readily available ;) (but not in this pr ofc)

sherifnada

Looks great! Just some small changes needed and question about unit tests

...rs/source-bigquery/src/main/java/io/airbyte/integrations/source/bigquery/BigQuerySource.java

airbyte-db/src/main/java/io/airbyte/db/bigquery/BigQueryDatabase.java

airbyte-db/src/main/java/io/airbyte/db/bigquery/BigQueryUtils.java

sherifnada · 2021-07-15T17:28:50Z

...-test/src/main/java/io/airbyte/integrations/standardtest/source/SourceComprehensiveTest.java

+   *
+   * @return formatted list of test cases
+   */
+  public String getMarkdownTestTable() {


this is great! next step will be write this into a markdown file as part of the build so it's readily available ;) (but not in this pr ofc)

airbyte-integrations/connectors/source-bigquery/src/main/resources/spec.json

airbyte-db/src/main/java/io/airbyte/db/bigquery/BigQueryUtils.java

subodh1810

Nice!

…rces/spec.json Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

…e-bigquery

…nto aleonets/1876-source-bigquery

vbhakta8 · 2021-07-20T21:05:30Z

airbyte-integrations/connectors/source-bigquery/src/main/resources/spec.json

+  "supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
+  "connectionSpecification": {
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "title": "BigQuery Destination Spec",


Should this be BigQuery Source Spec?

sherifnada · 2021-07-21T14:22:26Z

@heade you might need to make some changes similar to #4874 to make sure the config format matches spec.json

heade · 2021-07-22T09:39:42Z

/test connector=source-bigquery

🕑 source-bigquery https://github.com/airbytehq/airbyte/actions/runs/1055650710
✅ source-bigquery https://github.com/airbytehq/airbyte/actions/runs/1055650710

heade · 2021-07-22T14:17:14Z

/publish connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/1056455930
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/1056455930

DoNotPanicUA and others added 26 commits June 22, 2021 17:18

unfinished jdbcsource separation

286e327

creation AbstactRelation

6a62eaa

Migrate StateManager to new abstract level (JdbcSource -> RelationalS…

0aaf904

…ource)

fix imports

6dcedf5

move configs to Database level + fix MySql source

6d8e976

make in line jdbc source with a new impl

042c527

Fix ScaffoldJavaJdbcSource template

147e166

rename AbstractField to CommonField. Now it

246528a

s not an abstract class. + add default implementation for `AbstractRelationalDbSource.getFullyQualifiedTableName`

format

e8d29c9

rename generated files in line with their location

111abae

bonus renaming

3bc83b2

move utility methods specific for jdbc source to a proper module

18e86d8

internal review update

67f1e6a

BigQueryDatabase impl without row transformation

42eb2c5

add Static method for BigQueryDatabase instancing

9326d35

remove data type parameter limitation + rename class parameters

c7d2eff

Merge remote-tracking branch 'origin/aleonets/4024-abstract-source' i…

c78a441

…nto aleonets/1876-source-bigquery

Move DataTypeUtils from jdbs to common + impl basic types BigQueryUtils

06f5d13

Merge remote-tracking branch 'origin/master' into aleonets/4024-abstr…

d16f883

…act-source # Conflicts: # airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

make DB2 in line with new relational abstract classes

75e7c99

add missing import

3dc5383

cover all biqquery classes + add type transformation method from Stan…

fd60d14

…dardSQLTypeName to JsonSchemaPrimitive

Merge remote-tracking branch 'origin/master' into 4024-abstract-source

b3e0801

close unused connections

5421313

Merge branch 'master' into aleonets/1876-source-bigquery

22356d0

# Conflicts: # airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

Merge remote-tracking branch 'origin/aleonets/4024-abstract-source' i…

32b53ee

…nto aleonets/1876-source-bigquery

DoNotPanicUA requested a review from yaroslav-hrytsaienko July 1, 2021 10:24

github-actions bot added area/connectors Connector related issues area/protocol CDK Connector Development Kit labels Jul 1, 2021

DoNotPanicUA requested a review from yaroslav-hrytsaienko July 15, 2021 08:51

Merge branch 'master' into aleonets/1876-source-bigquery

fd33eed

yaroslav-hrytsaienko approved these changes Jul 15, 2021

View reviewed changes

DoNotPanicUA marked this pull request as ready for review July 15, 2021 16:31

DoNotPanicUA requested review from sherifnada and subodh1810 July 15, 2021 16:31

DoNotPanicUA added the lang/java label Jul 15, 2021

sherifnada approved these changes Jul 15, 2021

View reviewed changes

subodh1810 approved these changes Jul 15, 2021

View reviewed changes

DoNotPanicUA and others added 3 commits July 16, 2021 13:33

Update airbyte-integrations/connectors/source-bigquery/src/main/resou…

6a58540

…rces/spec.json Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

review update

d6053f9

Implement processing for arrays and structures

e545247

DoNotPanicUA requested a review from sherifnada July 16, 2021 16:31

DoNotPanicUA and others added 5 commits July 16, 2021 19:39

format

2fd9199

Merge remote-tracking branch 'origin/master' into aleonets/1876-sourc…

45c7f0c

…e-bigquery

Merge remote-tracking branch 'origin/aleonets/1876-source-bigquery' i…

ebce19c

…nto aleonets/1876-source-bigquery

added bigquery secrets

46f5b3e

added bigquery secrets

e493468

vbhakta8 reviewed Jul 20, 2021

View reviewed changes

sherifnada approved these changes Jul 21, 2021

View reviewed changes

heade added 2 commits July 22, 2021 09:42

spec fix

05067c3

test configs fix

449a0b5

heade merged commit 4c0d1a0 into master Jul 22, 2021

heade deleted the aleonets/1876-source-bigquery branch July 22, 2021 12:39

heade restored the aleonets/1876-source-bigquery branch July 22, 2021 14:07

This was referenced Aug 12, 2021

fix value extraction from JSON in source comprehensive test #5379

Merged

SourceComprehensiveTest is broken #5380

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Big Query #4457

🎉 New Source: Big Query #4457

DoNotPanicUA commented Jul 1, 2021 •

edited

Loading

yaroslav-hrytsaienko left a comment

yaroslav-hrytsaienko Jul 15, 2021

yaroslav-hrytsaienko Jul 15, 2021

sherifnada Jul 15, 2021

sherifnada left a comment

sherifnada Jul 15, 2021

subodh1810 left a comment

vbhakta8 Jul 20, 2021

sherifnada commented Jul 21, 2021

heade commented Jul 22, 2021 •

edited by github-actions bot

Loading

heade commented Jul 22, 2021 •

edited by github-actions bot

Loading

🎉 New Source: Big Query #4457

🎉 New Source: Big Query #4457

Conversation

DoNotPanicUA commented Jul 1, 2021 • edited Loading

What

Comprehensive tests

Recommended reading order

Pre-merge Checklist

yaroslav-hrytsaienko left a comment

Choose a reason for hiding this comment

yaroslav-hrytsaienko Jul 15, 2021

Choose a reason for hiding this comment

yaroslav-hrytsaienko Jul 15, 2021

Choose a reason for hiding this comment

sherifnada Jul 15, 2021

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Jul 15, 2021

Choose a reason for hiding this comment

subodh1810 left a comment

Choose a reason for hiding this comment

vbhakta8 Jul 20, 2021

Choose a reason for hiding this comment

sherifnada commented Jul 21, 2021

heade commented Jul 22, 2021 • edited by github-actions bot Loading

heade commented Jul 22, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Jul 1, 2021 •

edited

Loading

heade commented Jul 22, 2021 •

edited by github-actions bot

Loading

heade commented Jul 22, 2021 •

edited by github-actions bot

Loading