Issues getting JSON credentials to work in JAR file

[jsnowacki]

I am trying to download some files from Google Cloud Storage to HDFS in a Spark (2.0.2) driver and then process the files in Spark. Everything works fine executed locally but not if I compile it into fat jar and deploy on Spark in YARN client mode (driver works locally); in both cases pulling is from the same cloud storage bucket using the same JSON credentials file. When I use the credentials from InputStream:

val inputStream = this.getClass.getResourceAsStream("/GCCredentials.json")
val credentials = GoogleCredentials.fromStream(inputStream)
      .createScoped(Seq(StorageScopes.DEVSTORAGE_READ_ONLY).asJava)
val storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService

I gent the below error:

Exception in thread "main" java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment.  Please set a project ID using the builder.
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:283)
        at com.google.cloud.HttpServiceOptions.<init>(HttpServiceOptions.java:179)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:69)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:27)
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:64)

I also tried with GOOGLE_APPLICATION_CREDENTIALS set and

val credentials = GoogleCredentials.getApplicationDefault
      .createScoped(Seq(StorageScopes.DEVSTORAGE_READ_ONLY).asJava)
val storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService

In this case I get different error:

Exception in thread "main" java.lang.NoSuchMethodError: org.json.JSONTokener.<init>(Ljava/io/InputStream;)V
        at com.google.cloud.ServiceOptions.serviceAccountProjectId(ServiceOptions.java:452)
        at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:346)
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:281)
        at com.google.cloud.HttpServiceOptions.<init>(HttpServiceOptions.java:179)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:69)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:27)
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:64)

Just to clarify, the credentials JSON file and org.json package are both in the JAR. Also, org.json has this method and seems to be in the right version inside JAR.

The build is done via SBT. I use sbt-assenbly plugin to build fat jar. GC Java library is in version 0.8.0:

    "com.google.cloud" % "google-cloud" % "0.8.0",
    // tried with and without it but nothing changed
    "com.google.cloud" % "google-cloud-nio" % "0.8.0",

Also source and target is set to Java 8:

javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")

[val-litvak]

I am experiencing the latter issue. I use the GOOGLE_APPLICATION_CREDENTIALS environment variable and call StorageOptions.getDefaultInstance().getService() (I also ran into the same with LoggingOptions.getDefaultInstance().getService()

java.lang.NoSuchMethodError: org.json.JSONTokener.<init>(Ljava/io/InputStream;)V
        at com.google.cloud.ServiceOptions.serviceAccountProjectId(ServiceOptions.java:452)
        at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:346)
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:281)
        at com.google.cloud.HttpServiceOptions.<init>(HttpServiceOptions.java:179)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:69)
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:27)
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:64)
        at com.google.cloud.storage.StorageOptions.getDefaultInstance(StorageOptions.java:99)

I also checked added org.json as dependency in my pom.xml to see if a newer one (20160810) or the one that google-cloud-core project was using (20151123) and both are producing the same error as above.

I am using maven and org.codehaus.mojo exec-maven-plugin 1.2.1

I am going to try to use the builder approach and manually set the credential values in the meantime.

[val-litvak]

I figured it out. @jsnowacki , not sure if you are running into the same issue as me, but for me it turns out that I had a conflicting version of the org.json package.

If this helps, this is what I ran into: one of my dependencies (https://github.com/jimmoores/quandl4j) depends on jackson-datatype-json-org ( https://github.com/jimmoores/quandl4j/blob/master/pom.xml#L303 ) which depends on a very old version of org.apache.geronimo.bundles -- this package provides the old implementation of org.json.JSONTokener , this outdated class does not have the constructor that accepts an InputStream as the parameter, it supports String or Reader -- However.. because this class depends on the InputStream version of the constructor ( https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-core/src/main/java/com/google/cloud/ServiceOptions.java#L452 ) , a java.lang.NoSuchMethodError exception is thrown.

What I ended up doing is using a suggested work-around here:
jimmoores/quandl4j#22 (comment)

to make sure that I am not using an older version of org.json, I also added this to my pom.xml to make sure that only the latest version of org.json.JSONTokener is used, which includes the constructor with the InputStream as the parameter.

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.json</groupId>
        <artifactId>json</artifactId>
        <version>20160810</version>
        <scope>compile</scope>
      </dependency>
  </dependencies>
<dependencyManagement>

You probably have a similar problem with a different dependency in your project.

also mvn dependency:tree and looking at the .classpath file for conflicting versions were both super helpful.

(you may need to find alternative commands/examples for SBT)

Hopefully this helps!

[garrettjonesgoogle]

@jsnowacki can you confirm if @val-litvak 's suggestion works for you?

[jsnowacki]

@val-litvak @garrettjonesgoogle unfortunately this workaround does not work for me . It might be down to using SBT assembly correctly. Currently I did not have time to spent on this issue in more depth, but it is likely I will come back to it. The workaround I use at the moment is the import is done in Python, which is not ideal but works at the moment.

[lesv]

@garrettjonesgoogle FatJar's have known issues.

[garrettjonesgoogle]

@lesv I can't quite tell why you posted a link relating to FatJar on this particular issue.

[lesv]

It's looking like I was switching between issues and left the comment on the wrong one. Not quite sure how that happened.

[shinfan]

@garrettjonesgoogle It looks like a dependency issue since our json version is over 1 year old. Does it concern you to update our gcloud json version to the latest?

[garrettjonesgoogle]

@shinfan we should see what kind of changes to json that it implies.

[shinfan]

Marked as closed.

This issue is caused by a dependency conflict introduced by an older org.json package used by an external dependency (also known as dependency hell). I think it should be the customers' responsibility of managing their dependency tree and there is not much we can do in google-cloud-java to prevent such issue.