Exceptions on SQS result kill the http-client in the SDK

[FlorianO]

We encounter the situation that the SDK throws an exception while processing the result of calls on sqs. This leads to a unusable system for us.
Somehow the http-client in the SDK is dead from thereon and all further calls will die.
(Also calls to cloudwatch)

Unable to unmarshall response (ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.)
Caused by:
ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.
Here are some of our stack traces.

Reason: Error in permanent callback.
Unable to unmarshall response (ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.)
AmazonHttpClient.java:627   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonSQSClient.java:875    com.amazonaws.services.sqs.AmazonSQSClient.invoke
AmazonSQSClient.java:543    com.amazonaws.services.sqs.AmazonSQSClient.sendMessage

Caused by:
Reason: ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.
XMLStreamReaderImpl.java:219    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource
XMLStreamReaderImpl.java:189    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.
XMLInputFactoryImpl.java:277    com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl
XMLInputFactoryImpl.java:129    com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader
XMLInputFactoryImpl.java:78 com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLEventReader
StaxResponseHandler.java:89 com.amazonaws.http.StaxResponseHandler.handle
StaxResponseHandler.java:42 com.amazonaws.http.StaxResponseHandler.handle
AmazonHttpClient.java:604   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonSQSClient.java:875    com.amazonaws.services.sqs.AmazonSQSClient.invoke
AmazonSQSClient.java:543    com.amazonaws.services.sqs.AmazonSQSClient.sendMessage

and

Reason: Unable to unmarshall response (ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.)
AmazonHttpClient.java:627   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonCloudWatchClient.java:788 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke
AmazonCloudWatchClient.java:310 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData

Caused by:
Reason: ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.
XMLStreamReaderImpl.java:219    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource
XMLStreamReaderImpl.java:189    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.
XMLInputFactoryImpl.java:277    com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl
XMLInputFactoryImpl.java:129    com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader
XMLInputFactoryImpl.java:78 com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLEventReader
StaxResponseHandler.java:89 com.amazonaws.http.StaxResponseHandler.handle
StaxResponseHandler.java:42 com.amazonaws.http.StaxResponseHandler.handle
AmazonHttpClient.java:604   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonCloudWatchClient.java:788 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke
AmazonCloudWatchClient.java:310 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData

[wmatveyenko]

Hi @FlorianO,
Did you recently update your JDK to 7u45? Oracle recently turned on a totalEntitySizeLimit on entity expansions in a document. You can override that with system property: -DentityExpansionLimit=xxxx - but we are investigating why the problem has just started. If you can revert to a previous JDK update, then the problem should go away. We will keep this issue updated with what we discover.

[wmatveyenko]

Hi this appears to be a bug that was introduced with 7u45. See this Oracle forum post: https://forums.oracle.com/thread/2594170

[nezoat]

Hi. Although this is a change in behavior starting in JDK 7u45, there is still a problem with the AWS SDK in that experiencing this exception kills the SDK for all further calls. It would seem the AWS SDK still has a bug in the error handling, as the original poster described as well.

[david-at-aws]

The intended change in the JDK was to start enforcing an entity expansion limit per document - this prevents someone from OOMing your process by feeding it an XML document with an obscenely-large number of entities to expand. The SDK would have handled that just fine - the request with a "bad" response would fail with the exception seen above, but the process would keep on going and further calls to the client which received reasonable responses would succeed.

The bug in the JDK is that they ended up enforcing an entity limit per process. If your process parses multiple XML documents over time (for example, responses from multiple calls to an AWS services), they will all increment the same shared counter. Even if each document is valid on its own, eventually the shared counter will hit the limit and any further attempt to parse an XML document with even a single entity that needs to be expanded will fail.

There's nothing wrong with the SDK's error handling here - it's just that once your process reaches this limit, we'll get the same error from the XML parser we use every time. Everything should work fine when using a (future) version of the JDK that fixes this bug and correctly enforces the limit per document.

[nezoat]

David - thank you very much for this excellent clarification. It makes perfect sense and explains why my process is crashing over time, from accumulation of XML processing.

[rushimusmaximus]

This seems like a pretty major oversight on the JDK team's part. A bug has been opened for this issue: https://bugs.openjdk.java.net/browse/JDK-8028111

Hopefully they'll get it fixed soon.

[djmcdonald]

Is there a recommended value to set the entity expansion limit to? I've set it to both 100,000 and 1,000,000 and yet the issue comes back (after a period of time, for example three to four days when it was set to 1,000,000 (this is polling the queue 10 times a second)).

I'm using version 1.4.1 of the aws-java-sdk so I don't know if upgrading to the latest would help fix?

[rushimusmaximus]

The issue is in the 7u45 JDK, not in the AWS SDK. You can set the value to 0 to disable the expansion limit. See http://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html

[mckamey]

While the bug is in JDK, it is worth noting that the version of OpenJDK in the latest Amazon Linux AMIs include this bug. So the out of box experience for many AWS users is that the AWS client will fail after 64,000 XML readers are created.

[davidsulpy]

Is there a work around to this that anyone is aware of?

[rushimusmaximus]

Create file at $JAVA_HOME/jre/lib/jaxp.properties

containing: jdk.xml.entityExpansionLimit=0

[davidsulpy]

Awesome, exactly what I'm looking for.

[mckamey]

Alternatively:

System.setProperty("entityExpansionLimit", "0");

Or upgrade your JVM to one which fixes the glitch.

Note: that disabling the entity expansion limit (by setting to zero) does allow a maliciously constructed XML doc to wreak havoc.

[zackangelo]

@mckamey Just a note, the entityExpansionLimit is deprecated going forward. You should use the full jdk.xml.entityExpansionLimit property if you have a JDK that supports it.

The previous entityExpansionLimit property was renamed to jdk.xml.entityExpansionLimit. For backward compatibility, the previous name is still supported, but it is recommended that you use the new name since it is more consistent with the other JAXP properties.

http://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html

[mckamey]

@zackangelo good point if forward compatibility is needed. Although by the time it is actually removed, the bug in the JDK will be fixed so it will no longer be needed. I was fixing this on JDK 6 so I'm not sure if it supports the namespaced property or not. The docs were vague as to when the change occurred and I didn't want to sift through the source.