Exceptions on SQS result kill the http-client in the SDK #123

Closed
FlorianO opened this Issue Oct 18, 2013 · 15 comments

Comments

Projects
None yet
9 participants
@FlorianO

We encounter the situation that the SDK throws an exception while processing the result of calls on sqs. This leads to a unusable system for us.
Somehow the http-client in the SDK is dead from thereon and all further calls will die.
(Also calls to cloudwatch)

Unable to unmarshall response (ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.)
Caused by:
ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.
Here are some of our stack traces.

Reason: Error in permanent callback.
Unable to unmarshall response (ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.)
AmazonHttpClient.java:627   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonSQSClient.java:875    com.amazonaws.services.sqs.AmazonSQSClient.invoke
AmazonSQSClient.java:543    com.amazonaws.services.sqs.AmazonSQSClient.sendMessage

Caused by:
Reason: ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.
XMLStreamReaderImpl.java:219    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource
XMLStreamReaderImpl.java:189    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.
XMLInputFactoryImpl.java:277    com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl
XMLInputFactoryImpl.java:129    com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader
XMLInputFactoryImpl.java:78 com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLEventReader
StaxResponseHandler.java:89 com.amazonaws.http.StaxResponseHandler.handle
StaxResponseHandler.java:42 com.amazonaws.http.StaxResponseHandler.handle
AmazonHttpClient.java:604   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonSQSClient.java:875    com.amazonaws.services.sqs.AmazonSQSClient.invoke
AmazonSQSClient.java:543    com.amazonaws.services.sqs.AmazonSQSClient.sendMessage

and

Reason: Unable to unmarshall response (ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.)
AmazonHttpClient.java:627   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonCloudWatchClient.java:788 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke
AmazonCloudWatchClient.java:310 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData

Caused by:
Reason: ParseError at [row,col]:[1,1] Message: JAXP00010001: The parser has encountered more than "64000" entity expansions in this document; this is the limit imposed by the JDK.
XMLStreamReaderImpl.java:219    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.setInputSource
XMLStreamReaderImpl.java:189    com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.
XMLInputFactoryImpl.java:277    com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl
XMLInputFactoryImpl.java:129    com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader
XMLInputFactoryImpl.java:78 com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLEventReader
StaxResponseHandler.java:89 com.amazonaws.http.StaxResponseHandler.handle
StaxResponseHandler.java:42 com.amazonaws.http.StaxResponseHandler.handle
AmazonHttpClient.java:604   com.amazonaws.http.AmazonHttpClient.handleResponse
AmazonHttpClient.java:331   com.amazonaws.http.AmazonHttpClient.executeHelper
AmazonHttpClient.java:202   com.amazonaws.http.AmazonHttpClient.execute
AmazonCloudWatchClient.java:788 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke
AmazonCloudWatchClient.java:310 com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData
@wmatveyenko

This comment has been minimized.

Show comment
Hide comment
@wmatveyenko

wmatveyenko Oct 18, 2013

Contributor

Hi @FlorianO,
Did you recently update your JDK to 7u45? Oracle recently turned on a totalEntitySizeLimit on entity expansions in a document. You can override that with system property: -DentityExpansionLimit=xxxx - but we are investigating why the problem has just started. If you can revert to a previous JDK update, then the problem should go away. We will keep this issue updated with what we discover.

Contributor

wmatveyenko commented Oct 18, 2013

Hi @FlorianO,
Did you recently update your JDK to 7u45? Oracle recently turned on a totalEntitySizeLimit on entity expansions in a document. You can override that with system property: -DentityExpansionLimit=xxxx - but we are investigating why the problem has just started. If you can revert to a previous JDK update, then the problem should go away. We will keep this issue updated with what we discover.

@wmatveyenko

This comment has been minimized.

Show comment
Hide comment
@wmatveyenko

wmatveyenko Oct 18, 2013

Contributor

Hi this appears to be a bug that was introduced with 7u45. See this Oracle forum post: https://forums.oracle.com/thread/2594170

Contributor

wmatveyenko commented Oct 18, 2013

Hi this appears to be a bug that was introduced with 7u45. See this Oracle forum post: https://forums.oracle.com/thread/2594170

@nezoat

This comment has been minimized.

Show comment
Hide comment
@nezoat

nezoat Nov 1, 2013

Hi. Although this is a change in behavior starting in JDK 7u45, there is still a problem with the AWS SDK in that experiencing this exception kills the SDK for all further calls. It would seem the AWS SDK still has a bug in the error handling, as the original poster described as well.

nezoat commented Nov 1, 2013

Hi. Although this is a change in behavior starting in JDK 7u45, there is still a problem with the AWS SDK in that experiencing this exception kills the SDK for all further calls. It would seem the AWS SDK still has a bug in the error handling, as the original poster described as well.

@david-at-aws

This comment has been minimized.

Show comment
Hide comment
@david-at-aws

david-at-aws Nov 1, 2013

Contributor

The intended change in the JDK was to start enforcing an entity expansion limit per document - this prevents someone from OOMing your process by feeding it an XML document with an obscenely-large number of entities to expand. The SDK would have handled that just fine - the request with a "bad" response would fail with the exception seen above, but the process would keep on going and further calls to the client which received reasonable responses would succeed.

The bug in the JDK is that they ended up enforcing an entity limit per process. If your process parses multiple XML documents over time (for example, responses from multiple calls to an AWS services), they will all increment the same shared counter. Even if each document is valid on its own, eventually the shared counter will hit the limit and any further attempt to parse an XML document with even a single entity that needs to be expanded will fail.

There's nothing wrong with the SDK's error handling here - it's just that once your process reaches this limit, we'll get the same error from the XML parser we use every time. Everything should work fine when using a (future) version of the JDK that fixes this bug and correctly enforces the limit per document.

Contributor

david-at-aws commented Nov 1, 2013

The intended change in the JDK was to start enforcing an entity expansion limit per document - this prevents someone from OOMing your process by feeding it an XML document with an obscenely-large number of entities to expand. The SDK would have handled that just fine - the request with a "bad" response would fail with the exception seen above, but the process would keep on going and further calls to the client which received reasonable responses would succeed.

The bug in the JDK is that they ended up enforcing an entity limit per process. If your process parses multiple XML documents over time (for example, responses from multiple calls to an AWS services), they will all increment the same shared counter. Even if each document is valid on its own, eventually the shared counter will hit the limit and any further attempt to parse an XML document with even a single entity that needs to be expanded will fail.

There's nothing wrong with the SDK's error handling here - it's just that once your process reaches this limit, we'll get the same error from the XML parser we use every time. Everything should work fine when using a (future) version of the JDK that fixes this bug and correctly enforces the limit per document.

@nezoat

This comment has been minimized.

Show comment
Hide comment
@nezoat

nezoat Nov 1, 2013

David - thank you very much for this excellent clarification. It makes perfect sense and explains why my process is crashing over time, from accumulation of XML processing.

nezoat commented Nov 1, 2013

David - thank you very much for this excellent clarification. It makes perfect sense and explains why my process is crashing over time, from accumulation of XML processing.

@rushimusmaximus

This comment has been minimized.

Show comment
Hide comment
@rushimusmaximus

rushimusmaximus Nov 18, 2013

This seems like a pretty major oversight on the JDK team's part. A bug has been opened for this issue: https://bugs.openjdk.java.net/browse/JDK-8028111

Hopefully they'll get it fixed soon.

This seems like a pretty major oversight on the JDK team's part. A bug has been opened for this issue: https://bugs.openjdk.java.net/browse/JDK-8028111

Hopefully they'll get it fixed soon.

@djmcdonald

This comment has been minimized.

Show comment
Hide comment
@djmcdonald

djmcdonald Dec 2, 2013

Is there a recommended value to set the entity expansion limit to? I've set it to both 100,000 and 1,000,000 and yet the issue comes back (after a period of time, for example three to four days when it was set to 1,000,000 (this is polling the queue 10 times a second)).

I'm using version 1.4.1 of the aws-java-sdk so I don't know if upgrading to the latest would help fix?

Is there a recommended value to set the entity expansion limit to? I've set it to both 100,000 and 1,000,000 and yet the issue comes back (after a period of time, for example three to four days when it was set to 1,000,000 (this is polling the queue 10 times a second)).

I'm using version 1.4.1 of the aws-java-sdk so I don't know if upgrading to the latest would help fix?

@rushimusmaximus

This comment has been minimized.

Show comment
Hide comment
@rushimusmaximus

rushimusmaximus Dec 2, 2013

The issue is in the 7u45 JDK, not in the AWS SDK. You can set the value to 0 to disable the expansion limit. See http://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html

The issue is in the 7u45 JDK, not in the AWS SDK. You can set the value to 0 to disable the expansion limit. See http://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html

@mckamey

This comment has been minimized.

Show comment
Hide comment
@mckamey

mckamey Jan 10, 2014

While the bug is in JDK, it is worth noting that the version of OpenJDK in the latest Amazon Linux AMIs include this bug. So the out of box experience for many AWS users is that the AWS client will fail after 64,000 XML readers are created.

mckamey commented Jan 10, 2014

While the bug is in JDK, it is worth noting that the version of OpenJDK in the latest Amazon Linux AMIs include this bug. So the out of box experience for many AWS users is that the AWS client will fail after 64,000 XML readers are created.

@davidsulpy

This comment has been minimized.

Show comment
Hide comment
@davidsulpy

davidsulpy Jan 11, 2014

Is there a work around to this that anyone is aware of?

Is there a work around to this that anyone is aware of?

@rushimusmaximus

This comment has been minimized.

Show comment
Hide comment
@rushimusmaximus

rushimusmaximus Jan 12, 2014

Create file at $JAVA_HOME/jre/lib/jaxp.properties

containing: jdk.xml.entityExpansionLimit=0

Create file at $JAVA_HOME/jre/lib/jaxp.properties

containing: jdk.xml.entityExpansionLimit=0

@davidsulpy

This comment has been minimized.

Show comment
Hide comment
@davidsulpy

davidsulpy Jan 12, 2014

Awesome, exactly what I'm looking for.

Awesome, exactly what I'm looking for.

@mckamey

This comment has been minimized.

Show comment
Hide comment
@mckamey

mckamey Jan 12, 2014

Alternatively:

System.setProperty("entityExpansionLimit", "0");

Or upgrade your JVM to one which fixes the glitch.

Note: that disabling the entity expansion limit (by setting to zero) does allow a maliciously constructed XML doc to wreak havoc.

mckamey commented Jan 12, 2014

Alternatively:

System.setProperty("entityExpansionLimit", "0");

Or upgrade your JVM to one which fixes the glitch.

Note: that disabling the entity expansion limit (by setting to zero) does allow a maliciously constructed XML doc to wreak havoc.

@zackangelo

This comment has been minimized.

Show comment
Hide comment
@zackangelo

zackangelo Feb 10, 2014

@mckamey Just a note, the entityExpansionLimit is deprecated going forward. You should use the full jdk.xml.entityExpansionLimit property if you have a JDK that supports it.

The previous entityExpansionLimit property was renamed to jdk.xml.entityExpansionLimit. For backward compatibility, the previous name is still supported, but it is recommended that you use the new name since it is more consistent with the other JAXP properties.

http://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html

@mckamey Just a note, the entityExpansionLimit is deprecated going forward. You should use the full jdk.xml.entityExpansionLimit property if you have a JDK that supports it.

The previous entityExpansionLimit property was renamed to jdk.xml.entityExpansionLimit. For backward compatibility, the previous name is still supported, but it is recommended that you use the new name since it is more consistent with the other JAXP properties.

http://docs.oracle.com/javase/tutorial/jaxp/limits/limits.html

@mckamey

This comment has been minimized.

Show comment
Hide comment
@mckamey

mckamey Feb 10, 2014

@zackangelo good point if forward compatibility is needed. Although by the time it is actually removed, the bug in the JDK will be fixed so it will no longer be needed. I was fixing this on JDK 6 so I'm not sure if it supports the namespaced property or not. The docs were vague as to when the change occurred and I didn't want to sift through the source.

mckamey commented Feb 10, 2014

@zackangelo good point if forward compatibility is needed. Although by the time it is actually removed, the bug in the JDK will be fixed so it will no longer be needed. I was fixing this on JDK 6 so I'm not sure if it supports the namespaced property or not. The docs were vague as to when the change occurred and I didn't want to sift through the source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment