New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in stage [HttpRequestParser] #1484

Open
hamsterready opened this Issue Oct 17, 2017 · 18 comments

Comments

Projects
None yet
6 participants
@hamsterready
Copy link

hamsterready commented Oct 17, 2017

Hi, I am using akka-http/10.0.9 and on production I see following exceptions (not very often, probably every 500k requests):

Error in stage [HttpRequestParser]: 762 
java.lang.ArrayIndexOutOfBoundsException: 762
        at akka.http.impl.engine.parsing.HttpHeaderParser.parseHeaderValue(HttpHeaderParser.scala:194)
        at akka.http.impl.engine.parsing.HttpHeaderParser.parseHeaderLine(HttpHeaderParser.scala:130)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.parseHeaderLines(HttpMessageParser.scala:125)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseHeaderLines(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseMessage(HttpRequestParser.scala:77)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.startNewMessage(HttpMessageParser.scala:100)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.startNewMessage(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$startNewMessage$1.apply(HttpMessageParser.scala:101)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$startNewMessage$1.apply(HttpMessageParser.scala:101)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$continue$2.apply(HttpMessageParser.scala:264)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$continue$2.apply(HttpMessageParser.scala:264)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.run$1(HttpMessageParser.scala:64)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.parseBytes(HttpMessageParser.scala:78)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseBytes(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.parseSessionBytes(HttpMessageParser.scala:60)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseSessionBytes(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.onPush(HttpRequestParser.scala:55)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:747)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:649)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:471)
        at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:423)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:529)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:495)
        at akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:63)
        at kamon.trace.Tracer$.withContext(TracerModule.scala:58)
        at akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:62)
        at akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:45)
        at akka.actor.ActorCell.invoke(ActorCell.scala:488)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

and sometimes:

Error in stage [HttpRequestParser]: null 
java.nio.BufferOverflowException: null
        at java.nio.Buffer.nextPutIndex(Buffer.java:521)
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
        at akka.http.impl.engine.parsing.HttpHeaderParser$.akka$http$impl$engine$parsing$HttpHeaderParser$$scanHeaderValue(HttpHeaderParser.scala:557)
        at akka.http.impl.engine.parsing.HttpHeaderParser$ModeledHeaderValueParser.apply(HttpHeaderParser.scala:499)
        at akka.http.impl.engine.parsing.HttpHeaderParser.parseAndInsertHeader$1(HttpHeaderParser.scala:173)
        at akka.http.impl.engine.parsing.HttpHeaderParser.parseHeaderValue(HttpHeaderParser.scala:187)
        at akka.http.impl.engine.parsing.HttpHeaderParser.parseHeaderLine(HttpHeaderParser.scala:130)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.parseHeaderLines(HttpMessageParser.scala:125)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseHeaderLines(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.akka$http$impl$engine$parsing$HttpMessageParser$$parseHeaderLinesAux(HttpMessageParser.scala:170)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$parseHeaderLines$1.apply(HttpMessageParser.scala:131)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$parseHeaderLines$1.apply(HttpMessageParser.scala:131)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$continue$1.apply(HttpMessageParser.scala:263)
        at akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$continue$1.apply(HttpMessageParser.scala:263)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.run$1(HttpMessageParser.scala:64)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.parseBytes(HttpMessageParser.scala:78)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseBytes(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpMessageParser$class.parseSessionBytes(HttpMessageParser.scala:60)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseSessionBytes(HttpRequestParser.scala:44)
        at akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.onPush(HttpRequestParser.scala:55)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:747)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:649)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:471)
        at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:423)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:529)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:495)
        at akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at akka.kamon.instrumentation.ActorMonitors$$anon$1$$anonfun$processMessage$1.apply(ActorMonitor.scala:63)
        at kamon.trace.Tracer$.withContext(TracerModule.scala:58)
        at akka.kamon.instrumentation.ActorMonitors$$anon$1.processMessage(ActorMonitor.scala:62)
        at akka.kamon.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorInstrumentation.scala:45)
        at akka.actor.ActorCell.invoke(ActorCell.scala:488)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

and then lot of these:

Error in stage [HttpRequestParser]: null 
java.lang.ArrayIndexOutOfBoundsException: null
(no stack trace)

I was not able to find the offending request (I have tcpdump but nothing special out there, nginx which run in front does not report errors neither).

BTW, I have seen akka/akka#18397 and I will try to disable cache and see if it helps.

@jrudolph jrudolph added the 0 - new label Oct 17, 2017

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Oct 17, 2017

Thanks for the report, @hamsterready. It would indeed be good to have more information about what kind of requests were running but I understand they may be hard to find. In any case, it's pretty likely that this is a bug.

@hamsterready

This comment has been minimized.

Copy link

hamsterready commented Oct 18, 2017

I set akka.http.parsing.header-cache.default = 0 and no such errors since redeploy, but I need to wait for larger traffic (when EST timezone wakes up). Will let you know.

@hamsterready

This comment has been minimized.

Copy link

hamsterready commented Oct 18, 2017

Unfortunately it happened again, shall I set these to 0 as well:

      User-Agent = 32

?

@hamsterready

This comment has been minimized.

Copy link

hamsterready commented Oct 18, 2017

For the record, after I set akka.http.parsing.header-cache.default = 0 I have not seen first exception any more. The second one is still present (very rare, in fact only once).

@hamsterready

This comment has been minimized.

Copy link

hamsterready commented Oct 18, 2017

BTW, no more ArrayIndexOutOfBoundsException only BufferOverflowException.

@lukasz-golebiewski

This comment has been minimized.

Copy link

lukasz-golebiewski commented Mar 26, 2018

I am also getting the same errors, but I would not like to disable header caching as we need high throughput. Is there any way of suppressing this without disabling caching? Is there a fix planned? @jrudolph

@halilakgun

This comment has been minimized.

Copy link

halilakgun commented Jun 9, 2018

Any updates here?

@raboof

This comment has been minimized.

Copy link
Member

raboof commented Jun 11, 2018

@halilakgun I looked into it a bit but haven't been able to reproduce it yet, would be very interested!

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Sep 26, 2018

In #2001 the same index 762 was reported, interesting...

@stephane-debart

This comment has been minimized.

Copy link

stephane-debart commented Dec 4, 2018

Hi,
We get the same errors:

Error in stage [HttpRequestParser]: 762
java.lang.ArrayIndexOutOfBoundsException: 762
	at akka.http.impl.engine.parsing.HttpHeaderParser.parseHeaderValue(HttpHeaderParser.scala:194)
	at akka.http.impl.engine.parsing.HttpHeaderParser.parseHeaderLine(HttpHeaderParser.scala:130)
        ...

And

Error in stage [HttpRequestParser]: null
java.lang.ArrayIndexOutOfBoundsException

We get them in 2 distinct services with completely different requests (graphQL POST in one case, REST GET in the other service)

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Dec 6, 2018

@stephane-debart is that reproducible for you? Could you post the headers in question?

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Dec 6, 2018

Error in stage [HttpRequestParser]: 762 java.lang.ArrayIndexOutOfBoundsException: 762

Seems like an invariant that is stated in the scaladoc is violated:

The array has a fixed size of 254 rows (since we can address at most 256 - 1 values with the node MSB and we always have fewer branching nodes than values).

Needs more digging.

Error in stage [HttpRequestParser]: null

If it's the same exception as above at HeapByteBuffer.put then that's about weird Unicode content in a header triggering a bug.

@stephane-debart

This comment has been minimized.

Copy link

stephane-debart commented Dec 6, 2018

@jrudolph No unfortunately, we don't have a repro. It happens a few times every days on our production servers. And we don't know what are the corresponding headers.
Is there an easy way to get or log these headers before having this error ? As the stack trace doesn't contain any reference to our own code but just tells that we are inside an actor, I even don't really know what is the top level function involved...

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Dec 6, 2018

Is there an easy way to get or log these headers before having this error ?

No not yet, if we cannot figure out the issue otherwise, we'll add extra logging.

As the stack trace doesn't contain any reference to our own code but just tells that we are inside an actor, I even don't really know what is the top level function involved...

Yep, that's expected. It's in the code path for parsing an HTTP message from a client which happens asynchronously.

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Dec 6, 2018

Error in stage [HttpRequestParser]: null

If it's the same exception as above at HeapByteBuffer.put then that's about weird Unicode content in a header triggering a bug.

I think it can happen if data is coming in slices that cut a multi-byte UTF-8 sequence in the middle. Then byteAt will fail with a NotEnoughDataException but the ByteBuffer isn't reset.

jrudolph added a commit to jrudolph/akka-http that referenced this issue Dec 6, 2018

@stephane-debart

This comment has been minimized.

Copy link

stephane-debart commented Dec 6, 2018

@jrudolph I've been able to find in our front reverse proxy log the headers of one of the request which failed.
Unfortunately, headers added by intermediate proxies are not in this log. And at least one of these intermediate proxies may add unicode characters in headers (it's adding user name).
So would it help if I send this incomplete list of headers to you ?

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Dec 13, 2018

The java.nio.BufferOverflowException: null problem has been fixed in #2306.

@jrudolph

This comment has been minimized.

Copy link
Member

jrudolph commented Dec 13, 2018

So would it help if I send this incomplete list of headers to you ?

I guess it would have more helped with the Unicode issue which I already fixed in #2306. For the other issue, the problem might probably accrue over multiple requests made over the same connection, so we would probably need the full and exact set of headers for all requests or responses on one connection.

I guess it's reproducible by feeding the HeaderParser a big enough set of different enough headers so that the branching nodes are overflowing the fixed-size data structures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment