S3Client.download fails silently on 404

[eyalfa]

when attempting to download a missing key from s3, the returned ObjectMetaData future completes successfully and the returned Source completes without emitting any data.

I suspect the root cause is in akka.stream.alpakka.s3.impl.S3Stream#download:

def download(s3Location: S3Location,
               range: Option[ByteRange],
               sse: Option[ServerSideEncryption]): (Source[ByteString, NotUsed], Future[ObjectMetadata]) = {
    import mat.executionContext
    val s3Headers = S3Headers(sse.fold[Seq[HttpHeader]](Seq.empty) { _.headersFor(GetObject) })
    val future = request(s3Location, rangeOption = range, s3Headers = s3Headers)
    val source = Source
      .fromFuture(future.flatMap(entityForSuccess))
      .map(_.dataBytes)
      .flatMapConcat(identity)
    val meta = future.map(resp ⇒ computeMetaData(resp.headers, resp.entity))
    (source, meta)
  }

as you can see the initial future is an HttpResponse future, in this case it has a 404 status, it's first being flatMapped to create the source and later mapped to created the metadata object.
the flatMap internally checks the response status while map fails to do so.

in my use-case I pass this source to play's Ok.sendEntity (play.api.mvc.Results.Status#sendEntity) so I don't control on the way it is being consumed and how are failures handled, I'd prefer being signaled about the missing key (via a failure or 'Option[ObjectMetadata]') so my program can behave accordingly.

[2m]

Interesting. The Future is flatmapped with entityForSuccess which checks the response status code for success which is false for 404s and eventually throws. There is something more going on.

[eyalfa]

@2m , thanks for the quick reply.
please notice that the flatMap using entityForSuccess applies only for the returned source, the meta future is produed by mapping over the original response future and it does not check the status.
assuming I'd find some time in the weakend, I might be able to introduce a failing test and a fix PR.

[anikiforovopensource]

@eyalfa I've seen more silent failures with the alpakka-s3 connector.

When the processing is slow, and the s3 connector is back-pressured, it causes the source to hang irrecoverably. The connector wont throw an exception and wont emit any more data downstream. Eventually this causes the entire application to hang, as it cannot make any progress, and wont fail so it can be restarted.

[eyalfa]

@lexn82 , can u reproduce this behavior in a test?
This is quite surprising as the underlying stream is implemented by akka-http which is supposed to behave properly in case of backpreasure.

[2m]

Fixed in #1122