Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persistent caching of HTTP responses fails due to presence of non-serializable classes #137

Closed
grigoriy opened this issue Jun 15, 2017 · 5 comments
Closed

Persistent caching of HTTP responses fails due to presence of non-serializable classes #137

grigoriy opened this issue Jun 15, 2017 · 5 comments

Comments

@grigoriy
Copy link
Contributor

Play WS Version (2.5.x / etc)

1.0.0-M10

API (Scala / Java / Neither / Both)

Scala

Operating System (Ubuntu 15.10 / MacOS 10.10 / Windows 10)

Arch Linux (4.10.11-1-ARCH)

JDK (Oracle 1.8.0_72, OpenJDK 1.8.x, Azul Zing)

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Library Dependencies

"org.ehcache" % "ehcache" % "3.3.1"

Expected Behavior

Please describe the expected behavior of the issue, starting from the first action.

  1. send an HTTP request
  2. get an HTTP response
  3. cache the response to disk

Actual Behavior

  1. works fine
  2. works fine
  3. java.io.NotSerializableException is thrown:
[] INFO   - Ehcache key GET http://localhost:8181/greeting?name=Foo recovered from (o.e.c.i.r.LoggingRobustResilienceStrategy)
java.io.NotSerializableException: play.api.libs.ws.ahc.cache.CacheableHttpResponseBodyPart
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at scala.collection.immutable.List$SerializationProxy.writeObject(List.scala:476)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at org.ehcache.impl.serialization.CompactJavaSerializer.serialize(CompactJavaSerializer.java:94)
        ... 54 common frames omitted
Wrapped by: org.ehcache.spi.serialization.SerializerException: java.io.NotSerializableException: play.api.libs.ws.ahc.cache.CacheableHttpResponseBodyPart
        at org.ehcache.impl.serialization.CompactJavaSerializer.serialize(CompactJavaSerializer.java:100)
        at org.ehcache.impl.internal.store.offheap.portability.OffHeapValueHolderPortability.encode(OffHeapValueHolderPortability.java:52)
        at org.ehcache.impl.internal.store.offheap.portability.OffHeapValueHolderPortability.encode(OffHeapValueHolderPortability.java:31)
        at org.terracotta.offheapstore.storage.PortabilityBasedStorageEngine.writeMapping(PortabilityBasedStorageEngine.java:64)
        at org.terracotta.offheapstore.OffHeapHashMap.tryWriteEntry(OffHeapHashMap.java:703)
        at org.terracotta.offheapstore.OffHeapHashMap.writeEntry(OffHeapHashMap.java:687)
        at org.terracotta.offheapstore.OffHeapHashMap.computeWithMetadata(OffHeapHashMap.java:1947)
        at org.terracotta.offheapstore.AbstractLockedOffHeapHashMap.computeWithMetadata(AbstractLockedOffHeapHashMap.java:582)
        at org.terracotta.offheapstore.concurrent.AbstractConcurrentOffHeapMap.computeWithMetadata(AbstractConcurrentOffHeapMap.java:729)
        at org.ehcache.impl.internal.store.offheap.EhcacheConcurrentOffHeapClockCache.compute(EhcacheConcurrentOffHeapClockCache.java:152)
        at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1094)
        ... 44 common frames omitted
Wrapped by: org.ehcache.core.spi.store.StoreAccessException: org.ehcache.spi.serialization.SerializerException: java.io.NotSerializableException: play.api.libs.ws.ahc.cache.CacheableHttpResponseBodyPart
        at org.ehcache.core.exceptions.StorePassThroughException.handleRuntimeException(StorePassThroughException.java:70)
        at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.computeWithRetry(AbstractOffHeapStore.java:1109)
        at org.ehcache.impl.internal.store.offheap.AbstractOffHeapStore.put(AbstractOffHeapStore.java:316)
        at org.ehcache.impl.internal.store.tiering.TieredStore.put(TieredStore.java:135)
        at org.ehcache.core.Ehcache.put(Ehcache.java:198)
        at api.ProxyService$EhcacheHttpCache$1.put(HttpService.scala:69)
        at play.api.libs.ws.ahc.cache.AhcHttpCache.put(AhcHttpCache.scala:60)
        at play.api.libs.ws.ahc.cache.AhcHttpCache.cacheResponse(AhcHttpCache.scala:281)
        at play.api.libs.ws.ahc.cache.AsyncCachingHandler.processFullResponse(AsyncCachingHandler.scala:201)
        at play.api.libs.ws.ahc.cache.AsyncCachingHandler.onCompleted(AsyncCachingHandler.scala:142)
        at play.shaded.ahc.org.asynchttpclient.netty.NettyResponseFuture.getContent(NettyResponseFuture.java:193)
        at play.shaded.ahc.org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:227)
        at play.shaded.ahc.org.asynchttpclient.netty.handler.HttpHandler.finishUpdate(HttpHandler.java:58)
        at play.shaded.ahc.org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:159)
        at play.shaded.ahc.org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:187)
        at play.shaded.ahc.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:76)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at play.shaded.ahc.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at play.shaded.ahc.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at play.shaded.ahc.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
        at play.shaded.ahc.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
        at play.shaded.ahc.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
        at play.shaded.ahc.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at play.shaded.ahc.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
        at play.shaded.ahc.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:625)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:560)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:477)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:439)
        at play.shaded.ahc.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)

Reproducible Test Case

https://github.com/grigoriy/play-ws-persistent-cache-bug

Notes

The problem is with CacheableHttpResponseStatus and CacheableHttpResponseBodyPart which are not serializable.
@wsargent
Copy link
Member

wsargent commented Jun 15, 2017 •
edited
Loading

Play-WS does not assume that you want to use Java serialization as your persistence mechanism. In fact, persistence isn't handled through Play-WS -- that's abstracted out to the HTTP cache trait, which is responsible for serialization:

https://github.com/playframework/play-ws/blob/master/play-ahc-ws-standalone/src/main/scala/play/api/libs/ws/ahc/cache/Cache.scala

So the problem here is that the ehcache cache has overflow to disk set to true:

https://github.com/grigoriy/play-ws-persistent-cache-bug/blob/master/source/main/resources/ehcache.xml

Instead, you can configure ehcache as follows:

https://playframework.com/documentation/2.6.x/WsCache

<ehcache
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd"
        name="play-ws-cache"
        updateCheck="false"
        >

	<cache name="play-ws-cache" maxEntriesLocalHeap="10000" eternal="false"
		timeToIdleSeconds="360" timeToLiveSeconds="1000" overflowToDisk="false" />

</ehcache>

@grigoriy
Copy link
Contributor Author

grigoriy commented Jun 15, 2017 •
edited
Loading

Thanks for the explanation. Could you consider it as a feature request or is it completely out of the scope of Play-WS? I am willing to participate in implementing this if needed. The use case is as follows: we want to use long-term on-disk HTTP cache, adhering to RFC 5861, 7232 and 7234, which would allow our application to reuse it after the application restarts. Do you think it is feasible without (Java) serializability of responses?

@wsargent
Copy link
Member

wsargent commented Jun 15, 2017 •
edited
Loading

Oh, it's totally feasible. It's not something I can do myself at this point, but it's an option -- that being said, the assumption right now is that the cache is in-memory and synchronous. You can see in the README there's an example of Caffeine being used:

https://github.com/playframework/play-ws#caching

One thing that would have to be added is that Cache would have to implement Future[T] so that you could have IO thread handling behind an asynchronous boundary: #133

For serialization, I'd probably use Kryo with Chill and setRegistrationRequired: https://github.com/twitter/chill / https://github.com/EsotericSoftware/kryo#registration as it'd be faster and inline with the JVM. If you're looking for a shared cache between programs, then personally I think Avro is the best overall choice.

@wsargent
Copy link
Member

It'd be best if you could do it as an sbt subproject with a dependency on the AHC implementation. I didn't do any work to break out the cache implementation or API from AHC itself, so that's another possibility.

@grigoriy
Copy link
Contributor Author

Thanks for advice Will! I'll definitely consider the options you've mentioned. Closing. Cheers, Grigoriy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wsargent @grigoriy