Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recievign very large attachments in the clent will cause a OutOfMemoryError [SWS-707] #801

Closed
gregturn opened this issue Apr 29, 2011 · 10 comments

Comments

@gregturn
Copy link
Member

@gregturn gregturn commented Apr 29, 2011

Karthik Ramacahndran opened SWS-707 and commented

AbstractHttpSenderConnectio.hasResponse reads the enter message into memory in order to ensure a message exists. When dealing with large messages this causes an OutOfMemoryError:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:548)
at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:496)
at gov.ic.dia.cdir.client.CdirFetchServicesClientImpl.fetchWithAttachments(CdirFetchServicesClientImpl.java:160)
at gov.ic.dia.cdir.client.CommandLineClient.testFetchEnclosure(CommandLineClient.java:283)
at gov.ic.dia.cdir.client.CommandLineClient.main(CommandLineClient.java:129)


Affects: 1.5.9, 2.0.1

Attachments:

Referenced from: pull request #99, and commits cba6791

4 votes, 7 watchers

@gregturn
Copy link
Member Author

@gregturn gregturn commented Jun 9, 2011

Christopher Wong commented

This rather defeats the point of the various underlying streaming message facilities, such as StreamingWebServiceMessage or the Axiom message factory with payloadCaching disabled. It makes WebServiceTemplate unusable for large messages.

@gregturn
Copy link
Member Author

@gregturn gregturn commented Mar 18, 2014

Arjen Poutsma commented

AbstractHttpSenderConnection only reads the message to determine the content-length of the message. If a Content-Length header is set in the response, it is used and the message is not read.

@gregturn
Copy link
Member Author

@gregturn gregturn commented Apr 23, 2014

Gyula Szalai commented

How can I set the Content-Length header to the appropriate value on the server side? I tried it with an EndpointInterceptor, but I don't know how to get the full size of the response message in bytes.

@gregturn
Copy link
Member Author

@gregturn gregturn commented Apr 24, 2014

Gyula Szalai commented

I have a response that contains a 600MB attachment, since I was not able to calculate the full size of the response (see my previous comment) I just set the Content-Length to a relatively small value (10000) and tried the client. I got the following error:

java.lang.RuntimeException: org.jvnet.mimepull.MIMEParsingException: Reached EOF, but there is no closing MIME boundary.
	at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:223)
	at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:68)
	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:163)
	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:130)
	at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:198)
	at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:181)
	at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:106)
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parseAll(MimePullMultipart.java:122)
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parse(MimePullMultipart.java:133)
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(MimeMultipart.java:210)
	at com.sun.xml.messaging.saaj.soap.MessageImpl.initializeAllAttachments(MessageImpl.java:1444)
	at com.sun.xml.messaging.saaj.soap.MessageImpl.getAttachments(MessageImpl.java:950)
	at org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:363)
	at org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:323)
	at org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109)
	at org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:957)
	at com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:100)
	at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:75)
	at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:150)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:244)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
	at com.sun.xml.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:127)
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:324)
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:307)
	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127)
	at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:738)
	at org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62)
	at org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:409)
	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:598)
	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

Then I set the Content-Length header to a very large value (10,000,000,000) I get the following error (I think it's the same as the original):

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2271)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
	at org.springframework.util.StreamUtils.copy(StreamUtils.java:125)
	at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109)
	at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156)
	at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
	at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
	at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:591)
	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

I used an EndpointInterceptor implementation to set the Content-Lenght header as follows:

    ...

    @Override
    public boolean handleResponse(MessageContext messageContext, Object endpoint) throws Exception {
        WebServiceMessage responseMessage = messageContext.getResponse();
        AbstractSoapMessage abstractSoapMessage = (AbstractSoapMessage) responseMessage;
        SaajSoapMessage saajSoapMessage = (SaajSoapMessage) abstractSoapMessage;
        SOAPMessage soapMessage = saajSoapMessage.getSaajMessage();
        
        MimeHeaders mimeHeaders = soapMessage.getMimeHeaders();
        mimeHeaders.addHeader("Content-Length", "10000000000");
        return true;
    }

    ....

Am I doing something wrong?

@gregturn
Copy link
Member Author

@gregturn gregturn commented Apr 24, 2014

Marcus Kalthoff commented

I would try to set it to a value less than Integer.MAX_VALUE
2147483647 <-- max int
10000000000 <--- your value

@gregturn
Copy link
Member Author

@gregturn gregturn commented Apr 24, 2014

Gyula Szalai commented

Thanks Marcus, that helped. If I set it to 2147483647 I don't get OOM on the client anymore.
However, I think, an attachment (at least in theory) can be larger than 2 GB, so it can be a problem.

@gregturn
Copy link
Member Author

@gregturn gregturn commented Sep 4, 2015

Ralf Stuckert commented

If you do streaming, you often don't know any content-lenght in advance.

And you do not need to read the whole content into memory just to decide, whether there is content. This could easily be done using a PushbackInputStream, reading one byte and pushing it back.

@gregturn
Copy link
Member Author

@gregturn gregturn commented Sep 4, 2015

Ralf Stuckert commented

Implementation using a PushbackInputStream in order to avoid reading the whole content

@gregturn
Copy link
Member Author

@gregturn gregturn commented Sep 4, 2015

Ralf Stuckert commented

Here is an implemenation using a PushbackInputStream: [^PatchedAbstractHttpSenderConnection.java]

It works fine and allows streaming of large content without content-length header being set.

@gregturn
Copy link
Member Author

@gregturn gregturn commented Sep 4, 2015

Ralf Stuckert commented

And since the method hasResponse() is final, there is no chance to fix by inheritance.

@gregturn gregturn closed this Oct 24, 2017
@gregturn gregturn added this to the 3.0.0.RELEASE milestone Sep 22, 2020
@gregturn gregturn self-assigned this Sep 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant