[BEAM-13990] support date and timestamp fields #17404

reuvenlax · 2022-04-20T05:14:28Z

No description provided.

reuvenlax · 2022-04-20T05:19:00Z

@liu-du I copied your PR and made some changes to integrate it with recent changes - hope that's ok!

reuvenlax · 2022-04-20T05:19:09Z

R: @liu-du

codecov · 2022-04-20T06:57:59Z

Codecov Report

Merging #17404 (b11c07b) into master (e6aae70) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #17404   +/-   ##
=======================================
  Coverage   73.85%   73.85%           
=======================================
  Files         690      690           
  Lines       90843    90843           
=======================================
+ Hits        67092    67095    +3     
+ Misses      22538    22535    -3     
  Partials     1213     1213

Flag	Coverage Δ
python	`83.65% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/transforms/combiners.py	`93.03% <0.00%> (-0.39%)`	⬇️
.../python/apache_beam/ml/inference/sklearn_loader.py
...thon/apache_beam/ml/inference/sklearn_inference.py	`92.68% <0.00%> (ø)`
...eam/runners/interactive/interactive_environment.py	`90.49% <0.00%> (+0.30%)`	⬆️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (+0.74%)`	⬆️
sdks/python/apache_beam/utils/interactive_utils.py	`95.12% <0.00%> (+7.31%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6aae70...b11c07b. Read the comment docs.

reuvenlax · 2022-04-20T15:26:18Z

Run Java PreCommit

reuvenlax · 2022-04-20T16:21:47Z

Run Python PreCommit

reuvenlax · 2022-04-21T05:32:59Z

R: @yirutang

reuvenlax · 2022-04-25T21:10:32Z

friendly ping!

aaltay · 2022-04-28T17:47:25Z

@chamikaramj @yirutang - Could you please review this PR?

yirutang · 2022-04-28T22:53:14Z

...le-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java

@@ -523,14 +523,6 @@ public void testTimePartitioning() throws Exception {
    testTimePartitioning(method);
  }

-  @Test
-  public void testTimePartitioningStorageApi() throws Exception {


Why remove this?

AFAICT is was just duplicating the above test

yirutang · 2022-04-28T23:16:51Z

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java

@@ -84,10 +143,10 @@ public static class SchemaDoesntMatchException extends SchemaConversionException
          .put("NUMERIC", Type.TYPE_STRING) // Pass through the JSON encoding.
          .put("BIGNUMERIC", Type.TYPE_STRING) // Pass through the JSON encoding.
          .put("GEOGRAPHY", Type.TYPE_STRING) // Pass through the JSON encoding.
-          .put("DATE", Type.TYPE_STRING) // Pass through the JSON encoding.
+          .put("DATE", Type.TYPE_INT32)
          .put("TIME", Type.TYPE_STRING) // Pass through the JSON encoding.
          .put("DATETIME", Type.TYPE_STRING) // Pass through the JSON encoding.


TIME and DATETIME can be Int64 so that the encoding size is smaller. Shouldn't block this PR:
https://github.com/googleapis/java-bigquerystorage/blob/main/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java#L334

yirutang · 2022-04-28T23:23:51Z

...ud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java

+        if (value instanceof String) {
+          return value;
+        } else if (value instanceof BigDecimal) {
+          return ((BigDecimal) value).toPlainString();


We do provide encoding here:
https://github.com/googleapis/java-bigquerystorage/blob/main/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/BigDecimalByteStringEncoder.java

So that they can be pass in more efficiently. Not blocking this PR.

yirutang · 2022-04-28T23:27:35Z

...-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoIT.java

+          .set("timeValue", LocalTime.parse("00:52:07.123456"))
+          .set("datetimeValue", LocalDateTime.parse("2019-08-16T00:52:07.123456"))
+          .set("dateValue", LocalDate.parse("2019-08-16"))
+          .set("numericValue", new BigDecimal("23.4"))


bignumeric test?

int and double timestamp value test?

reuvenlax · 2022-04-29T21:07:34Z

@yirutang comments addressed

yirutang · 2022-04-29T22:33:53Z

...latform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProtoTest.java

+          .put("timestampvalue", 43L)
+          .put("timevalue", "00:52:07.123456")
+          .put("datetimevalue", "2019-08-16T00:52:07.123456")
+          .put("datevalue", (int) LocalDate.parse("2019-08-16").toEpochDay())
          .put("numericvalue", "23.4")


Could we test repeated byte string? I always had some problems with it.

@yirutang added extra test

reuvenlax · 2022-04-30T04:52:17Z

Run GoPortable PreCommit

…amp fields

yirutang · 2022-10-11T07:38:52Z

Sorry for the delay, will try to get to it today.

…

On Thu, Apr 28, 2022 at 10:47 AM Ahmet Altay ***@***.***> wrote: @chamikaramj <https://github.com/chamikaramj> @yirutang <https://github.com/yirutang> - Could you please review this PR? — Reply to this email directly, view it on GitHub <#17404 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGYVEUWXOABSWB6TZZH27TVHLFLRANCNFSM5T22M6VA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Thanks. Yiru

github-actions bot added gcp io java labels Apr 20, 2022

reuvenlax requested a review from chamikaramj April 20, 2022 05:19

github-actions bot added the build label Apr 20, 2022

reuvenlax mentioned this pull request Apr 20, 2022

[BEAM-13990] BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API #16926

Closed

4 tasks

reuvenlax mentioned this pull request Apr 28, 2022

Handle invalid rows in the Storage Api sink #17423

Merged

yirutang reviewed Apr 28, 2022

View reviewed changes

support date and timestamp fields

2774b8a

reuvenlax force-pushed the vortex_date_timestamp branch from 8195ce6 to 2774b8a Compare April 29, 2022 21:09

yirutang reviewed Apr 29, 2022

View reviewed changes

add tests

b11c07b

yirutang approved these changes Apr 30, 2022

View reviewed changes

reuvenlax merged commit 58b4d76 into apache:master Apr 30, 2022

jrmccluskey pushed a commit to jrmccluskey/beam that referenced this pull request May 3, 2022

Merge pull request apache#17404: [BEAM-13990] support date and timest…

4a266b8

…amp fields

damccorm mentioned this pull request Jun 5, 2022

Data type and conversions are stricter for BQ sinks #21625

Closed

This was referenced Nov 10, 2022

BigQueryIO cannot write to DATE and TIMESTAMP columns when using Storage Write API #21456

Closed

BigQuery Storage Write API fails with type mismatch error for DATE type #21327

Closed

ahmedabu98 mentioned this pull request Jul 26, 2023

[Bug]: Storage Write API fails on Batch Pipelines on 2.49 #27670

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-13990] support date and timestamp fields #17404

[BEAM-13990] support date and timestamp fields #17404

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 20, 2022

codecov bot commented Apr 20, 2022 •

edited

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 21, 2022

reuvenlax commented Apr 25, 2022

aaltay commented Apr 28, 2022

yirutang Apr 28, 2022

reuvenlax Apr 29, 2022

yirutang Apr 28, 2022

reuvenlax Apr 29, 2022

yirutang Apr 28, 2022

reuvenlax Apr 29, 2022

yirutang Apr 28, 2022

yirutang Apr 28, 2022

reuvenlax Apr 29, 2022

reuvenlax commented Apr 29, 2022

yirutang Apr 29, 2022

reuvenlax Apr 30, 2022

reuvenlax commented Apr 30, 2022

yirutang commented Oct 11, 2022 via email

[BEAM-13990] support date and timestamp fields #17404

[BEAM-13990] support date and timestamp fields #17404

Conversation

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 20, 2022

codecov bot commented Apr 20, 2022 • edited

Codecov Report

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 20, 2022

reuvenlax commented Apr 21, 2022

reuvenlax commented Apr 25, 2022

aaltay commented Apr 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reuvenlax commented Apr 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reuvenlax commented Apr 30, 2022

yirutang commented Oct 11, 2022 via email

codecov bot commented Apr 20, 2022 •

edited