jython 2.7.1 limits size of http 'post'

[kasemir]

About a month ago jython was updated from 2.7.0 to 2.7.1.
Since then, submitting scans fails when the size of the scan exceeds about 65k.

A display (BOY or Display Builder) that runs a script like this:

file = open("xml")
xml = file.read()
print xml

from scan.client.scanclient import ScanClient
scan_client = ScanClient()
result = scan_client.simulate(xml)

fails when the XML for the scan that it tries to submit exceeds about 65k.
The same script works fine when executed by plain python,
and it also works fine when using jython 2.7.0.

Need to either find the magic configuration parameter for jython 2.7.1 that is causing this limitation, or go back to jython 2.7.0.

[kasemir]

epics-rip/maven-osgi-bundles#74

[kasemir]

Same problem for BOY, Display Builder, and Phoebus as long as they use jython 2.7.1

[kasemir]

Standalone demo:

import logging, urllib2

xml = "<commands>\n"
for i in range(1000):
   xml += "      <comment>\n"
   xml += "        <text>Start Taking Data</text>\n"
   xml += "      </comment>\n"
xml += "</commands>\n"
print "XML: %d bytes" % len(xml)

logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger("_socket")
log.setLevel(level=logging.DEBUG)

# Post XML to URL
url = "http://localhost:4810/simulate"
req = urllib2.Request(url)
req.add_header('content-type' , 'text/xml')
opener = urllib2.build_opener()
print ("Posting...")
response = opener.open(req, xml)
print ("Done posting..")

result = response.read()
response.close()
print result

Works OK with C-Python and jython 2.7.0.

With jython 2.7.1, OK with range(900) (64823 bytes of XML),
but fails with range(1000) (72023 bytes of XML).

jython 2.7.1 is stuck in the opener.open(..) call that posts the XML.
It will log this:
DEBUG:_socket:Sending data <<<'POST /simulate HTTP>>>
DEBUG:_socket:Syncing on future DefaultChannelPromise@7e5efcab(incomplete) for send
DEBUG:_socket:Channel ready for write
DEBUG:_socket:Channel ready for write
DEBUG:_socket:Channel ready for write
DEBUG:_socket:Sent data <<<'POST /simulate HTTP>>>
DEBUG:_socket:Channel ready for write
DEBUG:_socket:Waiting on recv
DEBUG:_socket:head=None incoming=[]
.. which suggests that it wrote the data and is waiting for a reply.
On the scan server side, however, only 65363 bytes have been received,
still waiting for the rest of the 72023 bytes promised in the HTTP content-length header.

Wireshark also shows that jython sends four instances of a "TCP segment of a reassembled PDU",
then stops.

After 30 seconds, the scan server closes the connection (idle timeout).

[kasemir]

Fixed via update in PyScanClient that avoids the jython issue in urllib2 by instead using Java HTTP API,
PythonScanClient/PyScanClient@e5a9499

[kasemir]

Tried Jython 2.7.2a1+ built from tip of https://hg.python.org/jython.
Contains same urllib2, very similar urllib and _socket.
Updates Netty from 4.1.11 to 4.1.24.
Exhibits same problem.

[kasemir]

The truncation happens in jython's version of Lib/_socket.py.

In jython 2.7.0, the socket.send() method would simply send the data.
For Jython 2.7.1 it only sends the channel.bytesBeforeUnwritable():

    def send(self, data, flags=0):
        # FIXME this almost certainly needs to chunk things
...
        bytes_writable = self.channel.bytesBeforeUnwritable()
..
        sent_data = data[:bytes_writable]

        future = self.channel.writeAndFlush(Unpooled.wrappedBuffer(sent_data))

That bytes_writable happens to be 65536.
The FIXME has been there in the 2.7.0 version, which simply sent all the data but might have caused delays.
This commit removed the potential for blocking by simply truncating the data, without adding a mechanism that would later send the remaining data:
https://hg.python.org/jython/diff/f7edd144b85a/Lib/_socket.py

[kasemir]

Found a jython ticket: http://bugs.jython.org/issue2618