Unable to fully read images of an epub FXL

[johanpoirier]

I tried to open a comic book packaged into a fixed layout epub but each image fails to be displayed.
The image is very slowly drawn but is stopped in the middle of the process.

The device: OnePlus One (Android 6.0.1 / Cyanogen OS / 3Go RAM).
The app: r2-testapp 2.0.0-alpha.2

The stack trace:

10-29 09:17:18.789 5578-11276/org.readium.r2reader E/NanoHTTPD: Could not send response to the client
    java.io.IOException: Can't read ResourceInputStream
        at org.readium.r2.shared.fetcher.ResourceInputStream.read(ResourceInputStream.kt:107)
        at org.nanohttpd.protocols.http.response.Response.sendBody(Response.java:336)
        at org.nanohttpd.protocols.http.response.Response.sendBodyWithCorrectEncoding(Response.java:313)
        at org.nanohttpd.protocols.http.response.Response.sendBodyWithCorrectTransferAndEncoding(Response.java:300)
        at org.nanohttpd.protocols.http.response.Response.send(Response.java:268)
        at org.nanohttpd.protocols.http.HTTPSession.execute(HTTPSession.java:432)
        at org.nanohttpd.protocols.http.ClientHandler.run(ClientHandler.java:75)
        at java.lang.Thread.run(Thread.java:818)
     Caused by: org.readium.r2.shared.fetcher.Resource$Exception$Other: java.lang.Exception: Unable to skip enough bytes
        at org.readium.r2.shared.fetcher.Resource$Exception$Companion.wrap(Resource.kt:192)
        at org.readium.r2.shared.fetcher.ArchiveFetcher$EntryResource.read(ArchiveFetcher.kt:114)
        at org.readium.r2.shared.fetcher.ArchiveFetcher$EntryResource$read$1.invokeSuspend(ArchiveFetcher.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.readium.r2.shared.fetcher.ResourceInputStream.read(ResourceInputStream.kt:95)
        at org.nanohttpd.protocols.http.response.Response.sendBody(Response.java:336) 
        at org.nanohttpd.protocols.http.response.Response.sendBodyWithCorrectEncoding(Response.java:313) 
        at org.nanohttpd.protocols.http.response.Response.sendBodyWithCorrectTransferAndEncoding(Response.java:300) 
        at org.nanohttpd.protocols.http.response.Response.send(Response.java:268) 
        at org.nanohttpd.protocols.http.HTTPSession.execute(HTTPSession.java:432) 
        at org.nanohttpd.protocols.http.ClientHandler.run(ClientHandler.java:75) 
        at java.lang.Thread.run(Thread.java:818) 
     Caused by: java.lang.Exception: Unable to skip enough bytes
        at org.readium.r2.shared.extensions.InputStreamKt$readRange$2.invokeSuspend(InputStream.kt:70)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

I can provide the epub file in private.

[mickael-menu]

@johanpoirier Can you send me the EPUB on Slack? Thanks

[mickael-menu]

I couldn't reproduce the crash on my device, so I'm not sure about the stacktrace. It could be that the same input stream was read concurrently by the HTTP server, which would be a mistake.

The slow rendering is caused by the fact that the images are deflated inside the archives, which means that when requesting ranges, we have to read the entry again from the start for each chunk. This can be really slow for large files (here images of ~2MB). To alleviate this issue:

• authors should store binary files like images as store instead of deflate in the ZIP
• on the streamer side, we should read bigger chunks when random ranges are requested
  • this draft PR does it using buffered(): Add an Audiobook navigator based on MediaSession r2-streamer-kotlin#126

[johanpoirier]

I ran some tests with the optimizations done on the audio-navigator PR-123 and r2-streamer PR-126 .

Each image of the epub have a size of ~ 1.3MB

The default buffer size is 8 * 1024 (see PublicationResourceHandler).

The results:

• with default buffer size: the image display crashes early (and still slowly to be displayed on other devices)
• 128 * 1024: still crashes but display is faster
• 1024 * 1024: image display crashes near the end (when the buffer is out of data I presume) and display is fast
• 2048 * 1024: no crash and display is still fast

So if the buffer size is greater than the image size, the image is displayed correctly.

[mickael-menu]

Thanks for the feedback Johan.

So it looks like using a bigger buffer and maybe letting reading apps control its size could be useful. I can think of another optimization that could help: Since in this case the resource is read from the start to the end, we could cache an input stream to reuse it for the next byte range requests, if the range is after the input stream's current offset. At the moment each range request creates a new input stream which reads from the start for deflated resources: https://github.com/readium/r2-shared-kotlin/blob/f291944e4204b1c1288120be062f8d66cfffebc9/r2-shared/src/main/java/org/readium/r2/shared/util/archive/JavaZip.kt#L35-L45

Although the crash is still a mystery...

[mickael-menu]

@johanpoirier I implemented the solution I talked about in my last comment in this PR: readium/r2-shared-kotlin#140

Page loading is much faster although not instantaneous, and I think you shouldn't see the crash anymore because it goes through a different code path. Let me know if you find any issue.