Support Spark dynamic partition pruning #881
Conversation
|
/gcbrun |
Codecov ReportBase: 47.04% // Head: 46.69% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #881 +/- ##
==========================================
- Coverage 47.04% 46.69% -0.35%
==========================================
Files 161 161
Lines 6101 6151 +50
Branches 728 725 -3
==========================================
+ Hits 2870 2872 +2
- Misses 3013 3061 +48
Partials 218 218
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
![sonatype-lift[bot]](https://avatars.githubusercontent.com/in/9843?s=60&v=4){kind=small}
...gquery-connector-common/src/main/java/com/google/cloud/spark/bigquery/SparkBigQueryUtil.java
Outdated
Show resolved
Hide resolved
...gquery-connector-common/src/main/java/com/google/cloud/spark/bigquery/SparkBigQueryUtil.java
Outdated
Show resolved
Hide resolved
...-connector-common/src/main/java/com/google/cloud/bigquery/connector/common/BigQueryUtil.java
Show resolved
Hide resolved
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
...gquery-connector-common/src/main/java/com/google/cloud/spark/bigquery/SparkBigQueryUtil.java
Outdated
Show resolved
Hide resolved
|
/gcbrun |
| @@ -46,8 +48,10 @@ | |||
| SupportsReportStatistics, | |||
| SupportsQueryPushdown { | |||
|
|
|||
| private BigQueryDataSourceReaderContext ctx; | |||
| private InputPartition[] partitions; | |||
| protected final Logger logger = LoggerFactory.getLogger(getClass()); | |||
There was a problem hiding this comment.
It is used in the Spark 3.2 subclass. I took a a page from the Spark framework book where the superclass defines the logger for the hierarchy, but for each class the logger name is of that class (because it is not a static variable)
| @@ -31,7 +31,7 @@ | |||
| </dependency> | |||
| <dependency> | |||
| <groupId>org.apache.spark</groupId> | |||
| <artifactId>spark-avro_2.13</artifactId> | |||
| <artifactId>spark-avro_2.12</artifactId> | |||
There was a problem hiding this comment.
Why do we need to downgrade the dependency version?
There was a problem hiding this comment.
Because we need all the dependencies to have the same scala version
|
/gcbrun |
Based on PR #687
Co-Authored-By: Zhou Fang coufon@users.noreply.github.com