Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change default read format to Arrow #180

Merged
merged 20 commits into from
Jun 3, 2020
Merged

Change default read format to Arrow #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8aab4c1
move SchemaConverters to scala
Gaurangi94 May 20, 2020
4efed2d
move SchemaConverters to java
Gaurangi94 May 20, 2020
46cdcac
import required libraries
Gaurangi94 May 20, 2020
7790700
import required libraries
Gaurangi94 May 20, 2020
2325d92
resolve conflicts on schemaconverter
Gaurangi94 May 20, 2020
32ebdef
resolve conflicts on schemaconverter
Gaurangi94 May 20, 2020
126c807
Merge branch 'master' of github.com:Gaurangi94/spark-bigquery-connector
Gaurangi94 May 21, 2020
234a2f0
port avro and arrow binary iterators to java
Gaurangi94 May 21, 2020
f6a00c0
merge with upstream
Gaurangi94 May 21, 2020
5a3ee65
add unchecked IO exception
Gaurangi94 May 21, 2020
62abeb3
add unchecked IO exception
Gaurangi94 May 21, 2020
e9b1ae8
refactoring
Gaurangi94 May 21, 2020
7ef8cd9
refactor code
Gaurangi94 May 21, 2020
d742f5c
Merge branch 'master' of github.com:Gaurangi94/spark-bigquery-connector
Gaurangi94 May 26, 2020
03f1262
Merge branch 'master' of https://github.com/GoogleCloudDataproc/spark…
Gaurangi94 May 27, 2020
59bd9ea
fix readme file for partition type
Gaurangi94 May 28, 2020
e136422
fix readme file for partition type
Gaurangi94 May 28, 2020
45208d8
fix readme file
Gaurangi94 May 29, 2020
26ffeeb
Merge remote-tracking branch 'direct/master'
Gaurangi94 Jun 2, 2020
2cf3515
change default read format to avro
Gaurangi94 Jun 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion connector/src/main/scala/com/google/cloud/spark/bigquery/SparkBigQueryOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ object SparkBigQueryOptions {
val ReadDataFormatOption = "readDataFormat"
val ViewsEnabledOption = "viewsEnabled"

val DefaultReadDataFormat: DataFormat = DataFormat.AVRO
val DefaultReadDataFormat: DataFormat = DataFormat.ARROW
val DefaultFormat: String = "parquet"
val DefaultIntermediateFormat: IntermediateFormat =
IntermediateFormat(DefaultFormat, FormatOptions.parquet())
Expand Down
8 changes: 4 additions & 4 deletions connector/src/test/scala/com/google/cloud/spark/bigquery/SparkBigQueryOptionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ class SparkBigQueryOptionsSuite extends FunSuite {
}

test("data format - no value set") {
assertResult("AVRO") {
assertResult("ARROW") {
val options = SparkBigQueryOptions(
parameters,
Map.empty[String, String], // allConf
Expand All @@ -134,10 +134,10 @@ class SparkBigQueryOptionsSuite extends FunSuite {
}
}

test("Set Read Data Format as Arrow") {
assertResult("ARROW") {
test("Set Read Data Format as Avro") {
assertResult("AVRO") {
val options = SparkBigQueryOptions(
parameters + ("readDataFormat" -> "Arrow"),
parameters + ("readDataFormat" -> "Avro"),
Map.empty[String, String], // allConf
new Configuration,
new SQLConf,
Expand Down