Akka Http: Nginx proxy error when returning large json payloads

[cmbaxter]

As mentioned in this user group post:

https://groups.google.com/forum/#!topic/akka-user/aDwaI8Xia74

I'm running into an error using akka http with nginx in front of it as a reverse proxy when I return large (MBs) json payloads. The error I see in the nginx log is:

upstream prematurely closed connection while reading upstream

I created a very simple route that can reproduce this issue. That code is as follows:

  pathPrefix("api" / "proxytest"){
    (get & parameters('numberOfA.as[Int], 'chunk.as[Boolean])){ (aNum, chunk) =>  
      val bunchOfA = (for(i <- 1 to aNum) yield "a").mkString
      val json = s"""{"test": "$bunchOfA"}"""

      val entity = 
        if (chunk){
          val chunks = Source.single(json).mapConcat(_.grouped(1024*10).toList).map(HttpEntity.Chunk.apply(_))   
          HttpEntity.Chunked(ContentTypes.`application/json`, chunks)
        }
        else{
          HttpEntity(ContentTypes.`application/json`, json )
        }

      val response = HttpResponse(StatusCodes.OK, entity = entity)
      complete(response)
    }
  }

If I test this route with a get request passing in numberOfA as something small, say 500, and chunk as false everything works when I go through nginx. But if I pass in something large, like 2100000 with chunk set to false, I get that nginx error. If I pass in chunk=true, everything works. Also, if I skip nginx and go right to my server, everything works regardless of chunking and size. I also noticed that I always get this dead letter warning when I hit the nginx error:

[INFO] [01/20/2016 09:42:02.626] [default-akka.actor.default-dispatcher-4] [akka://default/system/IO-TCP/selectors/$a/1] Message [akka.io.SelectionHandler$ChannelWritable$] from Actor[akka://default/deadLetters] to Actor[akka://default/system/IO-TCP/selectors/$a/1#-229858999] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

This seems to imply that maybe the connection is closed prematurely in akka http before nginx thinks it's done with it.

I have also attached my nginx config (version 1.6.2). Let me know if you need anything more from me.
nginx.conf.txt

[drewhk]

Which version of Akka Http have you used?

[cmbaxter]

@drewhk, it's 2.0.1

[drewhk]

ok, I don't think it matters, but please try 2.0.2 just to be sure.

[cmbaxter]

@drewhk, actually, it looks like it does work with 2.0.2. Any thoughts on what was fixed in between 2.0.1 and 2.0.2 that resolved this?

[drewhk]

I don't really know. This is the milestone: https://github.com/akka/akka/issues?q=is%3Aissue+milestone%3Astream-http-2.0.2+is%3Aclosed

[drewhk]

I close it for now, but please reopen if you encounter this problem again (for example on the upcoming 2.4 version)

[dispalt]

I had this exact same problem for the record, Ill try with 2.0.2 and report back.

[drewhk]

Thanks! It is still a mystery what solved it, but at least if you can confirm it works with 2.0.2 that would help :)

[rucek]

Had the same issue in 2.0.1, seems to work in 2.0.2.

[drewhk]

It could have been an error that has been fixed in that version, but got not logged but turned into an RST on the TCP connection, closing it earlier than nginx expected from the length field.

[dispalt]

2.0.2 definitely fixed it. Would love to know what it was, too.

[drewhk]

Some dark secrets should better remain forgotten. Joking aside, my bet would be an exception that propagated as onError through the output side of the http stream, down to the TCP level where it turned into an abort, but was not logged. We fixed it so it will log after 2.4, so if the issue returns I hope we will see it properly.

[adarshaj]

I see this happening again with akka-http latest version 10.0.0 with json size in the range of 100KB. Is there any workaround for this?

The nginx logs just reports this message:

readv() failed (104: Connection reset by peer) while reading upstream

Which isn't very helpful. There's no log messages on akka-http project's console.

[mariszin]

We have noticed this issue as well. Request from the akka-http, without using nginx as a proxy, request succeeds. If we use nginx (we must use it to proxy other resources for our app), we get net::ERR_CONTENT_LENGTH_MISMATCH in Chrome, and a similar error in Firefox.
Is there any known workaround?

[luksow]

Noticed that as well with req of size ~700K. Akka HTTP 10.0.0.

[lomigmegard]

We stumbled on this issue. The connection is closed when a request is made through nginx (reverse-proxy) and the return entity is large (~800K).
Versions: Scala 2.11.8, Akka 2.4.14, Akka HTTP 10.0.0, Nginx 1.10.1.
From the nginx logs, the connection is closed by Akka after the data is transmitted.
From Akka logs, nothing visible :( no exceptions at all.

We didn't find any workaround but to have smaller responses.

[ktoso]

Have you run the system with debug logging on? Of not please do.

…

On Dec 23, 2016 09:52, "Lomig Mégard" ***@***.***> wrote: We stumbled on this issue. The connection is closed when a request is made through nginx (reverse-proxy) and the return entity is large (~800K). Versions: Scala 2.11.8, Akka 2.4.14, Akka HTTP 10.0.0, Nginx 1.10.1. From the nginx logs, the connection is closed by Akka after the data is transmitted. From Akka logs, nothing visible :( no exceptions at all. We didn't find any workaround but to have smaller responses. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19542 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHYk84cQwH5tN_VdiaH1_B1QkvYPQ9Kks5rK4vbgaJpZM4HIqL8> .

[lomigmegard]

We were already in DEBUG level for akka.*.
We enabled akka.io.tcp.trace-logging and obtained the following logs:

2016-12-23T13:33:22.692Z DEBUG a.i.TcpListener akka://xxx/system/IO-TCP/selectors/$a/3 - New connection accepted 
2016-12-23T13:33:22.693Z DEBUG a.i.SelectionHandler akka://xxx/system/IO-TCP/selectors/$a - Executing [WorkerForCommand(RegisterIncoming(java.nio.channels.SocketChannel[connected local=/10.0.3.100:7070 remote=/10.0.3.200:53729]),Actor[akka://xxx/system/IO-TCP/selectors/$a/3#2099263749],)]
2016-12-23T13:33:22.696Z DEBUG a.i.TcpIncomingConnection akka://xxx/system/IO-TCP/selectors/$a/17 - [Actor[akka://xxx/user/StreamSupervisor-1/$$d#-949355466]] registered as connection handler
2016-12-23T13:33:22.696Z DEBUG a.i.TcpIncomingConnection akka://xxx/system/IO-TCP/selectors/$a/17 - Read [275] bytes.
2016-12-23T13:33:23.224Z DEBUG a.i.TcpIncomingConnection akka://xxx/system/IO-TCP/selectors/$a/17 - Wrote [86880] bytes to channel
2016-12-23T13:33:23.224Z DEBUG a.i.TcpIncomingConnection akka://xxx/system/IO-TCP/selectors/$a/17 - Got Close command but write is still pending.
2016-12-23T13:33:23.225Z DEBUG a.i.TcpIncomingConnection akka://xxx/system/IO-TCP/selectors/$a/17 - Got Abort command. RESETing connection.
2016-12-23T13:33:23.225Z DEBUG a.i.SelectionHandler akka://xxx/system/IO-TCP/selectors/$a - received AutoReceiveMessage Envelope(Terminated(Actor[akka://xxx/system/IO-TCP/selectors/$a/17#1252778664]),Actor[akka://xxx/system/IO-TCP/selectors/$a/17#1252778664])

In Nginx logs:

readv() failed (104: Connection reset by peer) while reading upstream

When trying with working response we don't have the Got Close command but write is still pending and Got Abort command. RESETing connection.

It seems that nginx is very aggressive to close the connection early and Akka doesn't like it.

[gisql]

I have the same issue (the file doesn't need to be that big --- exceed 10 times net MTU and boom!). I did some testing on my own. It doesn't seem to be nginx problem. I tried other proxy with the same result. Exactly the same setup worked with previous version (the 'experimental').

Looking at TCP dump, I can see nginx behaving correctly: it's akka which sends ACK, RST packet for no good reason.

Since the problem seems to be on TCP level, I'm trying to establish if it's the akka-http or akka-actor (with the akka.io.* tools). I tried downgrading akka-* to 2.4.8, but akka-http doesn't like it: it uses OptionVal, which appears in 2.4.9. The latter version produces the same result.

I also tried to go the other way: akka-http: 10.0.1 and akka-* 2.4.16 --- no joy.

BTW, in order to reproduce the problem, you have to have a proxy and akka-http sitting on different boxes (2 docker containers isn't enough, mind).

[gisql]

OK, I believe that it is a bug in 'akka X'. As far as I can tell, if a client sends header Connection: close, akka pushes data to tcp channel and promptly closes the connection (without waiting for ACKs and such).

We've found a work-around using nginx. Basically, you have to force HTTP 1.1 with Connection: keep-alive. In order to do so, you need to:

  upstream your-service {
    keepalive 100;
    server akka-http-box:port;
  }

# (...)

  location /your/location/ {
    proxy_pass       http://your-service/;
    proxy_http_version 1.1;
    proxy_set_header Connection ""; # important bit: without it nginx will send "Connection: close" anyway
  }

Hope that helps someone out there...

[jrudolph]

Thanks for the information @gisql.

Please report any additional info at akka/akka-http#459. I think we have already identified the issue but didn't get yet to implement and test a fix so far.