Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add BigQuery TimePartitioning support, fix #1419 #1466

Merged
merged 1 commit into from Oct 17, 2018

Conversation

@nevillelyh
Copy link
Member

commented Oct 16, 2018

It's hard to test except maybe with integration tests though.

@codecov

This comment has been minimized.

Copy link

commented Oct 16, 2018

Codecov Report

Merging #1466 into master will decrease coverage by 0.1%.
The diff coverage is 12.5%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1466      +/-   ##
==========================================
- Coverage   79.05%   78.94%   -0.11%     
==========================================
  Files         170      170              
  Lines        5141     5148       +7     
  Branches      305      373      +68     
==========================================
  Hits         4064     4064              
- Misses       1077     1084       +7
Impacted Files Coverage Δ
...main/scala/com/spotify/scio/bigquery/package.scala 60% <0%> (-4.29%) ⬇️
...n/scala/com/spotify/scio/bigquery/BigQueryIO.scala 47.25% <0%> (-1.62%) ⬇️
...om/spotify/scio/bigquery/BigQuerySCollection.scala 57.14% <50%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 433dbf7...81e9cf1. Read the comment docs.

@@ -170,7 +173,8 @@ object BigQueryTable {
final case class WriteParam(schema: TableSchema,
writeDisposition: WriteDisposition,
createDisposition: CreateDisposition,
tableDescription: String)
tableDescription: String,
timePartitioning: TimePartitioning)

This comment has been minimized.

Copy link
@regadas

regadas Oct 17, 2018

Contributor

@nevillelyh should this be something like:

Suggested change
timePartitioning: TimePartitioning)
timePartitioning: TimePartitioning = None)

in case users use the write api directly?

This comment has been minimized.

Copy link
@nevillelyh

nevillelyh Oct 17, 2018

Author Member

I thought about it but the other ones like {write,create}Disposition don't have defaults either so I left it out for consistency. Chances are users shouldn't use these directly?

This comment has been minimized.

Copy link
@regadas

regadas Oct 17, 2018

Contributor

IRL: we will do a new revision round around the IO's to make sure params are consistent.

This comment has been minimized.

Copy link
@nevillelyh

nevillelyh Oct 17, 2018

Author Member

Filed #1471

@@ -328,7 +333,8 @@ object BigQueryTyped {

object Table {
final case class WriteParam(writeDisposition: WriteDisposition,
createDisposition: CreateDisposition)
createDisposition: CreateDisposition,
timePartitioning: TimePartitioning)

This comment has been minimized.

Copy link
@regadas

regadas Oct 17, 2018

Contributor

@nevillelyh same thing here?

@regadas regadas merged commit 4326f3e into master Oct 17, 2018

2 of 4 checks passed

codecov/patch 12.5% of diff hit (target 79.05%)
Details
codecov/project 78.94% (-0.11%) compared to 433dbf7
Details
ci/circleci: build_211 Your tests passed on CircleCI!
Details
ci/circleci: build_212 Your tests passed on CircleCI!
Details

@regadas regadas deleted the neville/bq branch Oct 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.