New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Command Part is unknown" error when calling Java methods using large byte arrays #109

Closed
JoshRosen opened this Issue Oct 27, 2012 · 9 comments

Comments

Projects
None yet
4 participants
@JoshRosen
Contributor

JoshRosen commented Oct 27, 2012

It looks like calls to methods that accept byte arrays may fail depending on the size of the arrays. For example:

from py4j.java_gateway import java_import, JavaGateway                          
gateway = JavaGateway.launch_gateway(die_on_exit=True)                          
# This call works:                                                              
gateway.jvm.java.nio.ByteBuffer.wrap(bytearray(255 for _ in range(10)))         
# This call fails:                                                              
gateway.jvm.java.nio.ByteBuffer.wrap(bytearray(255 for _ in range(100)))

The second call fails with this error:

py4j.protocol.Py4JError: An error occurred while calling z:java.nio.ByteBuffer.wrap. Trace:
py4j.Py4JException: Command Part is unknown: /////////////////////////////////////////////////////////w==
    at py4j.Protocol.getObject(Protocol.java:290)
    at py4j.commands.AbstractCommand.getArguments(AbstractCommand.java:81)
    at py4j.commands.CallCommand.execute(CallCommand.java:77)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:680)

The issue seems to be related to the size of the arrays. I turned on debug logging and it looks like this might have something to do with how the byte array data is split across multiple lines in the text protocol.

@bartdag

This comment has been minimized.

Owner

bartdag commented Oct 28, 2012

Are you using the latest release (https://github.com/bartdag/py4j/tree/master/current-release) or the official one on Pypi/sourceforge?

@JoshRosen

This comment has been minimized.

Contributor

JoshRosen commented Oct 28, 2012

I'm using a a copy of the master branch (as of commit 3dbf380) that I compiled myself. This branch contains all of the commits in current-release, plus the launch_gateway() function that I added for launching a default Java gateway process from Python.

I was not able to reproduce this issue with the tagged 0.7 branch.

JoshRosen added a commit to JoshRosen/py4j that referenced this issue Oct 28, 2012

Fix line-wrapping in bytearray b64 encoding (see bartdag#109).
Extra newlines were being inserted to line-wrap the encoded strings to
76 characters. AbstractCommand.getArguments() expects arguments to appear
on a single line, so this resulted in the line-wrapped lines being interpreted
as separate arguments.
@bartdag

This comment has been minimized.

Owner

bartdag commented Oct 28, 2012

Thanks a lot for the pull request. I merged it and the tests are passing (I have two failures, but they seem to be unrelated to the pull request and I just updated my OS so I believe this is a problem on my side).

Thanks also for the easy to reproduce test case. Byte array have been a huge pain since I tried to support them. I hope this is the last issue with this ;-)

Can I close the issue?

@JoshRosen

This comment has been minimized.

Contributor

JoshRosen commented Oct 28, 2012

Yes, this fixes my issue. Thanks!

@JoshRosen JoshRosen closed this Oct 28, 2012

bartdag pushed a commit that referenced this issue May 27, 2016

Fix line-wrapping in bytearray b64 encoding (see #109).
Extra newlines were being inserted to line-wrap the encoded strings to
76 characters. AbstractCommand.getArguments() expects arguments to appear
on a single line, so this resulted in the line-wrapped lines being interpreted
as separate arguments.
@dhwajraj

This comment has been minimized.

dhwajraj commented Sep 9, 2016

This issue reappears in 0.10.3
works fine when I downgraded to 0.9

@bartdag

This comment has been minimized.

Owner

bartdag commented Sep 9, 2016

Hi @dhwajraj, does it fail with the code submitted by Josh at the top of the issue? When I try it with 0.10.3, I don't receive any error.

@dhwajraj

This comment has been minimized.

dhwajraj commented Sep 10, 2016

Hi @bartdag.yes the command error is different. I am getting "yro4" with 0.10.3

@bartdag

This comment has been minimized.

Owner

bartdag commented Sep 10, 2016

Can you open a new issue with some code examples to replicate? yr04 is a response formatted using Py4J text protocol so you should never see it. Also, make sure that both the Java and Python sides are using the same version of Py4J: the protocol significantly changed between 0.9 and 0.10.x.

@xinyicheng

This comment has been minimized.

xinyicheng commented Jan 12, 2018

Hi @bartdag ,When I try it with 0.10.6,it fails with this error:
py4j.protocol.Py4JJavaError: An error occurred while calling o22.awaitTermination. : py4j.Py4JException: Command Part is unknown: yro52 at py4j.Protocol.getObject(Protocol.java:307) at py4j.Protocol.getReturnValue(Protocol.java:438) at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:113) at com.sun.proxy.$Proxy15.call(Unknown Source)
How to fix the issue?thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment