S3 Download of file larger than 8388608 bytes throws EntityStreamSizeException

[thepont]

Using the Java DSL attempting to download a file larger then 8388608 bytes returns the following exception.

I have set my akka.http.[server|client].parsing.max-content-length setting to a higher setting, although I am wondering if there is a way to use the s3client without a size limit, such as the withoutSizeLimit directive.

2017-05-15 12:15:34 <> ERROR ProxyRequestInvocationHandler:43 - EntityStreamSizeException: actual entity size (Some(8466195)) exceeded content length limit (8388608 bytes)! You can configure this by setting `akka.http.[server|client].parsing.max-content-length` or calling `HttpEntity.withSizeLimit` before materializing the dataBytes stream.
[ERROR] [05/15/2017 12:15:34.637] [default-akka.actor.default-dispatcher-5] [akka://default/user/StreamSupervisor-1/flow-802-0-unknown-operation] Error during preStart in [akka.http.scaladsl.model.HttpEntity$Limitable@3c2b4414]
EntityStreamSizeException: actual entity size (Some(8466195)) exceeded content length limit (8388608 bytes)! You can configure this by setting `akka.http.[server|client].parsing.max-content-length` or calling `HttpEntity.withSizeLimit` before materializing the dataBytes stream.
	at akka.http.scaladsl.model.HttpEntity$Limitable$$anon$1.preStart(HttpEntity.scala:607)
	at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:520)
	at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:380)
	at akka.stream.impl.fusing.ActorGraphInterpreter.tryInit(ActorGraphInterpreter.scala:538)
	at akka.stream.impl.fusing.ActorGraphInterpreter.finishShellRegistration(ActorGraphInterpreter.scala:580)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:595)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:619)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:497)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:529)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Code used to create the issue.

   @PostConstruct
    void init(){
        BasicCredentials credentials = new BasicCredentials(awsKey, awsSecret);
        alpakkaS3Client = new akka.stream.alpakka.s3.javadsl.S3Client(credentials, awsRegion, system, materializer);
        s3Client = new AmazonS3Client();
    }

    public class S3DownloadFlow {
        String bucketName;
        String keyName;

        public S3DownloadFlow(String bucketName, String keyName){
            this.bucketName = bucketName;
            this.keyName = keyName;
        }

        public Flowable<String> download(){
            if(bucketName != null && keyName != null) {
                return Flowable.fromPublisher(
                        alpakkaS3Client.download(bucketName, keyName)
                                .map(ByteString::utf8String)
                                .runWith(Sink.asPublisher(AsPublisher.WITHOUT_FANOUT), materializer)
                );
            }
            else{
                return Flowable.empty();
            }
        }
    }

[johanandren]

Note that it should be akka.http.client.parsing.max-content-length for the client, the [server|client] is because the size limit stage does know if it is running on the server or the client.

[thepont]

@johanandren thanks, changed that for the client and it works for my purposes, also noticed this can be set to infinite to completly remove the restrictions.

I wonder if other users would like a more granular approch since they might be using the HTTP client for multiple requests with different levels of trust and expecting different size responses.

[johanandren]

I think that discussions belong in the Akka HTTP issue tracker, please open a ticket there about that (you can refer to this issue from that like this: #303)

[matsluni]

I believe there is already a more granular approach to set the size limit per request/response if I understand the docs correct: https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/scala/akka/http/scaladsl/model/HttpEntity.scala#L135.

Maybe this ticket can be closed than?

[eyalfa]

hi guys, I see this is long closed now but...
I hit the same issue, increasing the limit in the cfg does solve the issue, I'm just not feeling comfortable lifting this limit for all http streams on my application, does it make sense to add an alpaka/s3 specific configuration for this? or alternatively expose it via the api?

[ceecer1]

Hi guys, I am having the same issues as @eyalfa recently. In routes, we can use "withSizeLimit(Long.MaxValue)" but is there anything specific to alpakka/s3 ?

[anikiforovopensource]

Any objections to simply using HttpEntity.withoutSizeLimit for S3 calls?

[DanieleSassoli]

Hi guys, I'm experiencing the same issue for GCS connector... I believe the fix would be very similar to this, if so I can go ahead and raise a PR to fix this?
For the time being I'm thinking of working around this with what's suggested here

[seglo]

@DanieleSassoli please go ahead and create a PR. Thanks!