Skip to content
This repository has been archived by the owner on Dec 15, 2021. It is now read-only.

saveAsBigQueryTable() method returns NoSuchMethodError #62

Open
pranay29 opened this issue Mar 5, 2018 · 14 comments
Open

saveAsBigQueryTable() method returns NoSuchMethodError #62

pranay29 opened this issue Mar 5, 2018 · 14 comments

Comments

@pranay29
Copy link

pranay29 commented Mar 5, 2018

I am running my spark-shell with Scala on version 2.2.1

spark-shell --packages com.spotify:spark-bigquery_2.10:0.2.0

scala> sqlContext.setGcpJsonKeyFile(file_path)

scala> sqlContext.setBigQueryProjectId("proj")

scala> sqlContext.setBigQueryGcsBucket("dummy_bucket")

scala> sqlContext.setBigQueryDatasetLocation("US")

I am trying to load some data in BigQuery which returns an error as shown below--

scala> val df = Seq((1,1,1), (2,2,2)).toDF("A","B","C")

scala> df.show
+---+---+---+
|  A|  B|  C|
+---+---+---+
|  1|  1|  1|
|  2|  2|  2|
+---+---+---+


scala> df.saveAsBigQueryTable("proj:dataset_name.table_name")
java.lang.NoSuchMethodError: com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List;
  at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase$ParentTimestampUpdateIncludePredicate.create(GoogleHadoopFileSystemBase.java:572)
  at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.createOptionsBuilderFromConfig(GoogleHadoopFileSystemBase.java:1890)
  at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1587)
  at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:793)
  at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:756)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
  at org.apache.spark.sql.execution.datasources.DataSource.writeInFileFormat(DataSource.scala:394)
  at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:471)
  at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:217)
  at com.databricks.spark.avro.package$AvroDataFrameWriter$$anonfun$avro$1.apply(package.scala:26)
  at com.databricks.spark.avro.package$AvroDataFrameWriter$$anonfun$avro$1.apply(package.scala:26)
  at com.spotify.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:159)
  at com.spotify.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:171)
  ... 50 elided


Kindly help !!

@chitralverma
Copy link

Hi @pranay29 did you get around this ?

@pranay29
Copy link
Author

Hi @chitralverma ,

I still have not got around this. I don't know if its fixed yet.

@nevillelyh
Copy link
Contributor

It's the classic guava version conflict. You can probably workaround it by pinging guava and/or other dependency versions.

@chitralverma
Copy link

chitralverma commented Apr 17, 2018 via email

@pranay29
Copy link
Author

I am not sure if I change the existing Spark version since its in production environment.

@sunilpashikanti
Copy link

My spark verion is 2.2.1 run in to tis error kindly help

spark-shell --packages com.github.samelamin:spark-bigquery_2.11:0.2.4

import com.samelamin.spark.bigquery._

df.saveAsBigQueryTable("myproj:dataset.target1")
java.lang.NoSuchMethodError: com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(Ljava/lang/String;)Lcom/google/api/services/bigquery/model/TableReference;
at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:40)
... 50 elided

@mikerlt
Copy link

mikerlt commented Sep 17, 2019

has anyone found a work around for this?

@sunilpashikanti
Copy link

sunilpashikanti commented Sep 17, 2019 via email

@mikerlt
Copy link

mikerlt commented Sep 17, 2019

Ok, and what did you do to get it to work?

I have found it. Year ago

On Tue, Sep 17, 2019, 10:06 PM mikerlt @.***> wrote: has anyone found a work around for this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#62?email_source=notifications&email_token=AIMRWXAJQMJ2GFGBLVWPJSTQKEBSBA5CNFSM4ETSGEC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65EFLY#issuecomment-532300463>, or mute the thread https://github.com/notifications/unsubscribe-auth/AIMRWXF2RFLJ72MYWEKI3LLQKEBSBANCNFSM4ETSGECQ .

@sunilpashikanti
Copy link

sunilpashikanti commented Sep 18, 2019 via email

@mikerlt
Copy link

mikerlt commented Sep 18, 2019 via email

@sunilpashikanti
Copy link

sunilpashikanti commented Sep 18, 2019 via email

@sunilpashikanti
Copy link

sunilpashikanti commented Sep 18, 2019 via email

@mikerlt
Copy link

mikerlt commented Sep 18, 2019 via email

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants