Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google App Engine Support for Future #8362

Closed
scabug opened this issue Mar 6, 2014 · 10 comments

Comments

@scabug
Copy link

commented Mar 6, 2014

When I try to use Scala's scala.concurrent.Future on Google App Engine, it fails because access to sun.misc.Unsafe is blocked by App Engine's security manager. This is reproducible using the App Engine Java SDK without uploading anything to Google, or needing a Google account:

HTTP ERROR 500

Problem accessing /test. Reason:

    INTERNAL_SERVER_ERROR

Caused by:

java.lang.ExceptionInInitializerError
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:26)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.<init>(Future.scala:20)
	at scala.concurrent.impl.Future$.apply(Future.scala:30)
	at scala.concurrent.Future$.apply(Future.scala:492)
	at zappenginefutures.TestServlet.doGet(TestServlet.scala:20)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:487)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.ExceptionInInitializerError
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:32)
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:24)
	... 41 more
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
	at java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.security.AccessController.checkPermission(Unknown Source)
	at java.lang.SecurityManager.checkPermission(Unknown Source)
	at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:431)
	at java.lang.SecurityManager.checkPackageAccess(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.ClassLoader.checkPackageAccess(Unknown Source)
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:22)
	... 42 more

Caused by:

java.lang.ExceptionInInitializerError
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:32)
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.<init>(Future.scala:20)
	at scala.concurrent.impl.Future$.apply(Future.scala:30)
	at scala.concurrent.Future$.apply(Future.scala:492)
	at zappenginefutures.TestServlet.doGet(TestServlet.scala:20)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:487)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
	at java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.security.AccessController.checkPermission(Unknown Source)
	at java.lang.SecurityManager.checkPermission(Unknown Source)
	at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:431)
	at java.lang.SecurityManager.checkPackageAccess(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.ClassLoader.checkPackageAccess(Unknown Source)
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:22)
	... 42 more

Caused by:

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
	at java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.security.AccessController.checkPermission(Unknown Source)
	at java.lang.SecurityManager.checkPermission(Unknown Source)
	at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:431)
	at java.lang.SecurityManager.checkPackageAccess(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.ClassLoader.checkPackageAccess(Unknown Source)
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:22)
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.<init>(Future.scala:20)
	at scala.concurrent.impl.Future$.apply(Future.scala:30)
	at scala.concurrent.Future$.apply(Future.scala:492)
	at zappenginefutures.TestServlet.doGet(TestServlet.scala:20)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:487)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

The basic code I'm using to reproduce this is below, my actual code was somewhat more complicated and involved blocking HTTP access in my threads:

package zappenginefutures

import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit

import scala.concurrent.ExecutionContext
import scala.concurrent.Future

import com.google.appengine.api.ThreadManager

import javax.servlet.http.HttpServlet
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse

class TestServlet extends HttpServlet {
  override def doGet(req: HttpServletRequest, resp: HttpServletResponse) {
    implicit val executionContext = ExecutionContext.fromExecutorService(
      Executors.newFixedThreadPool(4, ThreadManager.currentRequestThreadFactory))

    val f = Future {
      Thread.sleep(1000L)
    }

    executionContext.shutdown()
    executionContext.awaitTermination(10, TimeUnit.SECONDS)
    
    resp.getWriter.println("OK")
  }
}
@scabug

This comment has been minimized.

Copy link
Author

commented Mar 6, 2014

Imported From: https://issues.scala-lang.org/browse/SI-8362?orig=1
Reporter: Carey Evans (carey)
Affected Versions: 2.11.0-RC1, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.11.4, 2.11.5, 2.11.6

@scabug

This comment has been minimized.

Copy link
Author

commented Mar 6, 2014

@adriaanm said (edited on Mar 6, 2014 7:17:02 PM UTC):
I'd recommend asking on Stack Overflow or scala-user for work arounds, as we don't really support GAE (although we'd like to).

[EDIT: please ignore the rest of this comment -- it was more of a stream of consciousness than anything else]

My un-educated guess is that this is because we ship with our own copy of the fork join implementation, since we target Java 6. It looks like GAE requires using the one that comes with the/their jdk. I'll ask around for a work around.

Could you try this suggestion from SO? http://stackoverflow.com/a/20627692/276895

Specifically, create an execution context that uses the jdk's fork join. It would be better if this just worked, though.

implicit val execCtx = scala.concurrent.ExecutionContext fromExecutorService new java.util.concurrent.ForkJoinPool

Sorry, I see you're already doing this. Missed that line…

--

Looks like my diagnosis was wrong again, it's another use of Unsafe in AbstractPromise, as you say in the title. I'll go have some more coffee now and leave it to the experts.

@scabug

This comment has been minimized.

Copy link
Author

commented Mar 6, 2014

@adriaanm said (edited on Mar 6, 2014 9:47:28 PM UTC):
Looks like it regressed in scala/scala#454 -- see comments there for potential improvements in 2.11.1

@scabug

This comment has been minimized.

Copy link
Author

commented Mar 6, 2014

Carey Evans (carey) said:
I tried replacing scala.concurrent.util.FuturePromise with one that accesses sun.misc.Unsafe.compareAndSwapObject() officially via AtomicReferenceFieldUpdater, rather than unofficially via scala.concurrent.util.Unsafe, and that works fine in the App Engine development server:

package scala.concurrent.impl;

import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;

public abstract class AbstractPromise {
	protected final static AtomicReferenceFieldUpdater<AbstractPromise, Object> updater = AtomicReferenceFieldUpdater
			.newUpdater(AbstractPromise.class, Object.class, "_ref");

	private volatile Object _ref;

	protected final Object getState() {
		return _ref;
	}

	protected final boolean updateState(Object oldState, Object newState) {
		return updater.compareAndSet(this, oldState, newState);
	}
}

Presumably the current code is a micro-optimisation that makes a noticeable difference even after Hotspot has inlined all the method calls. It would be nice if Scala could fall back to the standard Java API instead of raising java.lang.ExceptionInInitializerError when the internal Java classes are not accessible.

@scabug

This comment has been minimized.

Copy link
Author

commented Mar 8, 2014

Carey Evans (carey) said:
I've pushed a change to carey/scala@72e1927 that should fix this with no change to performance in the normal case. The backward-compatibility check in the Ant tests doesn't understand that C:\ is an absolute path on Windows, though, so I'll need to get it going in Linux to test it.

@scabug

This comment has been minimized.

Copy link
Author

commented Aug 5, 2014

@gkossakowski said:
The 2.11.2 is out so I'm rescheduling the issue for 2.11.3.

@scabug

This comment has been minimized.

Copy link
Author

commented Nov 4, 2014

@retronym said:
Updating fix-by version to 2.11.5.

@scabug

This comment has been minimized.

Copy link
Author

commented Feb 24, 2015

@scabug

This comment has been minimized.

Copy link
Author

commented Jun 19, 2015

@SethTisue said:
newer PR is scala/scala#4443

@scabug

This comment has been minimized.

Copy link
Author

commented Jul 24, 2015

Bart Jenkins (bauhaus9) said:
Yes, please fix this issue. It is stopping me from using Autowire with Scala-JS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.