PubSub: NullPointerException at com.google.apphosting.runtime.ApiProxyImpl$CurrentRequestThreadFactory.newThread(ApiProxyImpl.java:1267) #2275
Comments
|
After rolling back to the |
|
In the end I wasn't able to find a workaround, so I had to revert all my recent changes. In a separate test environment (Java 8 standard) I noticed a few other errors that may be relevant in addition to the original |
|
Also posted this to StackOverflow: https://stackoverflow.com/questions/45330377/pubsub-in-app-engine-standard-java-8-nullpointerexception-at-apiproxyimplcurre |
|
Found a mention of a similar problem here: https://groups.google.com/d/msg/google-appengine/2UTg4Eqskyk/PbqYEK3GAAAJ I tried the work-around suggested there (https://gist.github.com/cmaan/7752e3c4fd0b1ba90a745cb6db232206) and the NPE has stopped happening. But now I'm back to the other |
|
I see you found my crazy workaround in https://gist.github.com/cmaan/7752e3c4fd0b1ba90a745cb6db232206 We stumbled upon the same This way, we got Spanner and PubSub to work on java8. |
|
Thanks for the tip, @cmaan. I'm still having some trouble getting it working, though. What credentials provider did you use? Did you have to pass it in a file? What I tried to do was to get the default credentials from AppEngine before applying the work-around, kind of like this: That didn't seem to work, though. I'm still getting the same Maybe I'll have to put some service account credentials in a file in GCS and then load them in from there? |
|
@garrettjonesgoogle @pongad Do you two see any reason why this might happen? |
|
@jbaldassari One possible workaround is to use executor for Pub/Sub, as mentioned in #2150 . I think when gRPC request is initiated through executor (i.e., not in request thread), GrpcUtils will pick the right threadFactory. My guess is right now it picks requestThreadFactory, and new threads generated by this factory will be scoped to the request. So it will throw NPE. |
There might be a better way, but that's exactly what we currently do |
|
@neozwu I tried the workaround suggested in #2150, but I'm still getting the same NPE at Am I supposed to be using a normal Java I wonder if I also need to construct the |
|
Update on workaround testing:
|
|
I just tried another workaround which got me past both the Here's what the code looks like: Update: |
tcoffee-google
added
api: pubsub
|
@rrch per your #1752 (comment), is there any update on thread execution in GAE standard? |
|
@neozwu It seems some shared grpc thread factories are actually decided by classloading side effects. I guess that's the reason why this demo [1] works on GAE j8, while almost every other realistic setup won't. Regarding [1] https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/appengine-java8/spanner |
|
@cmaan thank you for your investigation, and your demo project. Let me summarize what I understand, and please feel free to correct me. So it seems (1) when a Grpc call is initiated in an appengine request, the underlying grpc library will pick up (2) But if the Grpc call is in a default background thread (by, for example, using executorService in appengine request), the underlying guava code will fail to invoke appengine api (even though the sys prop is set) due to appengine restrictions, and thus will use defaultThreadFactory, therefore Grpc will get defaultThreadFactory. @jbaldassari would you mind sharing some sample code, to help further investigating this issue? |
|
@neozwu sure, I can give you a couple of examples. First, some context. We have an interface called The following implementation does not work and results in NPE @ ApiProxyImpl.java:1267 : https://gist.github.com/jbaldassari/cf1160c7ef9207e6ddd562dc2a1951db Here's another example that does not work, but for a different reason. Here we're creating the Finally, a working example: https://gist.github.com/jbaldassari/3f9a93e8e956f16f5bad5d658d204e4c Here we're retrieving the credentials from the AppEngine request thread but creating the Hopefully these examples help. The key seems to be that in order for authentication to work the credentials must be retrieved from an AppEngine request thread, but in order to avoid the NPE the |
|
Take a look at https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/appengine-java8/gaeinfo (enable TOKEN) and you can get a token using the Metadata server from any thread. |
|
In gRPC, |
|
@ejona86 I'm not very familiar with gRPC codebase, but it seems to me regardless of |
|
@neozwu, ah, you're right. However, |
|
@ejona86 Yes, the restricted behavior for request threads makes sense. However, |
|
@neozwu, because GAE Java 8 didn't have that restriction. If it has that restriction, then at some point that changed. At the very least, we know it isn't as restricted as J7 because in J8 you can load binaries where in J7 you couldn't. Is there some documentation that describes the current limitations across all the GAE varieties? |
|
@ejona86 : confirmed with @ifigotin offline, on GAE J8, threads created by |
|
But if you create them as a normal java thread, they will. (at least on Java 8) |
|
@lesv exactly, I think that's why using ExectuorService will work - a normal java thread cannot access appengine api, so the underlying guava code thinks it's not on appengine, and dispatches to gRPC a defaultThreadFactory, which creates threads that outlive request's lifecycle. |
|
Why not use normal threads, and use the metadata server to get tokens? The ExecutorService really looks bad and isn't easy. |
|
@ejona86 , we need some kind of resolution for thread treatment by grpc in GAE J8. |
|
Should work with latest pub sub library. |
After upgrading my PubSub client to
0.20.3-beta(as well as a few other Google dependencies) and switching from the Java8 flexible environment to the Java8 standard environment I'm no longer able to publish messages to my PubSub topic. I'm seeing this error in the logs for my AppEngine standard Java8 environment after publishing a message andget()ing theFuture:I'm using a Gradle build with the following relevant dependencies:
The deployment environment is AppEngine standard java8. I haven't been able to find the source code for
ApiProxyImpl.javaanywhere, so I'm unable to do much more debugging here. I guess I'll try downgrading the dependency and hope it works.The text was updated successfully, but these errors were encountered: