OutOfMemoryError when downloading a large publication

[mickael-menu]

When attempting to download a large publication protected with LCP (> 50MB on my device), it fails with an OutOfMemoryError. It seems to be thrown by Fuel but could originate from FuelPromiseExtension.kt in r2-shared-kotlin.

2020-05-14 19:18:56.481 5860-5990/org.readium.reader W/org.readium.reader: Throwing OutOfMemoryError "Failed to allocate a 134217744 byte allocation with 25165824 free bytes and 121MB until OOM, target footprint 98989272, growth limit 201326592" (VmSize 5552448 kB)
2020-05-14 19:18:56.482 5860-5990/org.readium.reader W/System.err: java.lang.OutOfMemoryError: Failed to allocate a 134217744 byte allocation with 25165824 free bytes and 121MB until OOM, target footprint 98989272, growth limit 201326592
2020-05-14 19:18:56.482 5860-5990/org.readium.reader W/System.err:     at java.util.Arrays.copyOf(Arrays.java:3161)
2020-05-14 19:18:56.482 5860-5990/org.readium.reader W/System.err:     at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
2020-05-14 19:18:56.483 5860-5990/org.readium.reader W/System.err:     at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
2020-05-14 19:18:56.483 5860-5990/org.readium.reader W/System.err:     at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
2020-05-14 19:18:56.483 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.fuel.util.InputStreamsKt.copyTo(InputStreams.kt:14)
2020-05-14 19:18:56.484 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.fuel.core.requests.DownloadTaskRequest.call(DownloadTaskRequest.kt:19)
2020-05-14 19:18:56.484 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.fuel.core.DeserializableKt$response$result$1.invoke(Deserializable.kt:82)
2020-05-14 19:18:56.484 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.fuel.core.DeserializableKt$response$result$1.invoke(Unknown Source:0)
2020-05-14 19:18:56.485 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.result.Result$Companion.of(Result.kt:116)
2020-05-14 19:18:56.485 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.fuel.core.DeserializableKt.response(Deserializable.kt:82)
2020-05-14 19:18:56.485 5860-5990/org.readium.reader W/System.err:     at com.github.kittinunf.fuel.core.Request.response(Request.kt:326)
2020-05-14 19:18:56.485 5860-5990/org.readium.reader W/System.err:     at org.readium.r2.shared.FuelPromiseExtensionKt$promise$1.invoke(FuelPromiseExtension.kt:21)
2020-05-14 19:18:56.486 5860-5990/org.readium.reader W/System.err:     at org.readium.r2.shared.FuelPromiseExtensionKt$promise$1.invoke(Unknown Source:0)
2020-05-14 19:18:56.486 5860-5990/org.readium.reader W/System.err:     at nl.komponents.kovenant.TaskPromise$wrapper$1.invoke(promises-jvm.kt:138)
2020-05-14 19:18:56.486 5860-5990/org.readium.reader W/System.err:     at nl.komponents.kovenant.TaskPromise$wrapper$1.invoke(promises-jvm.kt:130)
2020-05-14 19:18:56.487 5860-5990/org.readium.reader W/System.err:     at nl.komponents.kovenant.NonBlockingDispatcher$ThreadContext.run(dispatcher-jvm.kt:327)
2020-05-14 19:18:56.487 5860-5990/org.readium.reader W/System.err:     at java.lang.Thread.run(Thread.java:919)

[mickael-menu]

It looks like it's fixed in Fuel 2.x: kittinunf/fuel#564

[mickael-menu]

I actually removed Fuel in this PR, because it was simpler to use the native HttpURLConnection rather than upgrade all the R2 libs to Fuel 2.

However this issue is a moving target... After fixing it in LCP, it appeared in the Format sniffers, because we're loading the full byte array in memory to sniff the content. I limited the in-memory sniffing to 100KB to fix this. But then, the same error popped up from AndroidPdfViewer which is also trying to load the full PDF in memory to be presented. To be continued...