Data is corrupted when transferred as byte arrays from Java

[agronholm]

I was baffled when my PDFs from JasperReports showed up blank. After a lot of head scratching, I found that if I wrote the PDF data to disk on the Java side, it was fine. But if I transferred the data as byte[] over the network link, it would get "wrinkles" on the way, rendering it useless. Wrinkles here meaning that certain bytes (0xD8-0xDF range) would instead be replaced with nulls (0x00). It took me one long sleepless night to track down this problem, but I finally nailed it. It happens on the Java side of things, when the response object is being written to the network socket. The UTF-8 encoder does not like certain characters which happen to be on the reserved range for UTF-16 surrogates which should always come in pairs. This causes the encoder to silently replace some characters with question marks, which in turn either silently corrupts the data or causes an exception on the Python side on decoding. I concluded that this was a protocol design flaw.

More information at: http://en.wikipedia.org/wiki/Mapping_of_Unicode_characters#Surrogates

[bartdag]

Thanks Alex for this bug report. You are right that on the Python side, this works just fine (I can send a bytearray containing 0xD8 to Java without any problem), but on the Java side, something goes wrong with the UTF-8 conversion for these bytes. I'm not sure whether it's Java that is trying to do too much or if it's Python that is not doing enough.

I made some small tests on Java and every time, I could convert 0xD8 using Py4J's protocol without problem, but when put together with the Python side on a socket, something went awfully wrong.

I'll boost the priority on this one byte[] support is obviously broken :-(

[agronholm]

I have verified that the problem lies entirely on the Java side -- I dove deep into the sun.* classes with the debugger and found that StreamEncoder is doing character replacing due to failed UTF-8 encoding of some characters.

[bartdag]

Yes, you are right. I found more discussion on StackOverflow and there is not way the current solution will work (http://stackoverflow.com/questions/965838/handling-unicode-surrogate-values-in-java-strings).

This is quite a bummer because there was a lot of work put behind the automated conversion of byte array :-( If I want to pass the byte array by value, I'll have to use a more basic approach (length of array followed by the raw bytes).

Not sure when I'll have the time to work on this one, but as soon as I can dedicate some time to Py4J, i'll work on this bug.

[agronholm]

Wasn't this fixed in 0.7? Why is it still open?

[bartdag]

Although I merged the pull request (post 0.7), the tests still fail for python 3.x so I there is still some work to do.

[agronholm]

Nice timing :)
I was JUST writing a bytearray-using application on Python 3.2 (chart generation with JFreeChart).

[bartdag]

Hi Alex, I finally fixed the remaining issues with byte array.

I had to expand the compatibility layer to make it work with Python 3 and I added the conversion of byte array between from Python to Java. The changes in the Python 3 libraries are subtle and it's not always clear when a function accepts/returns a character string or a byte string :-(

I'll have to do some cleanup, but all the tests are passing now so you can use the code in the master branch.

[agronholm]

Too bad py4j isn't installable directly from git due to no setup.py being present on the top level.

[agronholm]

I can probably test this again when you've reverted the listening behavior to what it was in 0.7. Or would say that listening only on the public ip of the machine is correct...?