New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to execute compiled script with JEP and Python 3.5.3 #77

Closed
pjr1060 opened this Issue May 6, 2017 · 6 comments

Comments

Projects
None yet
3 participants
@pjr1060

pjr1060 commented May 6, 2017

I have implemented a compile step to convert .py to .pyc files prior to execution. Each script is cached and if it is present it is used instead of the interpreted .py.

Everything works great when JEP is asked to execute the .pyc when on Python 2.7.13. However when I change environment and use Python 3.5.3 I get the following error...

Bad code object in .pyc file

Note I am aware of pyc magic numbers - the cached pyc files are cleaned and compiled with 3.5.3. Looking at pyembed.c I am getting past the magic number validation at line 1328 and seeing the error at the validation step at line 1334.

Here's the bit I can't understand. If I execute the compiled .pyc file using python3 directly it runs just fine.

I have attached a simple example that highlights the problem, the zip contains the following...

jepHello.py - native script
jepHello-2.7.13.pyc - compiled with 2.7.13 - runs from JEP on Python 2.7.13 and native python2
jepHello-3.5.3.pyc - compiled with 3.5.3 - fails with JEP on Python 3.5.3 but runs with native python3

jepHello.zip

Please let me know if there's something I'm missing or if this is a bug and needs more investigation I'd be very happy to help. JEP is an awesome piece of work!

Incidentally I am using JEP as the caching compiler too using the following ...

jep.eval("import py_compile");
String py="py_compile.compile(file='"+scriptFile.getAbsolutePath()+"', cfile='"+compiledFile.getAbsolutePath()+"' )";
jep.eval(py);
jep.eval(null); // flush

This works flawlessly and the JEP generated .pyc runs fine on any version of native python.

Thanks for any help you can offer...

Peter

@bsteffensmeier

This comment has been minimized.

Show comment
Hide comment
@bsteffensmeier

bsteffensmeier May 10, 2017

Member

I am not very familiar with this code but I think I might have it figured out. The comment on pyembed_run_pyc makes it clear that the code was gratuitously copyied from pythonrun.c::run_pyc_file. I compared the code in run_pyc_file in python 2.7 with the code in run_pyc_file in python 3.5. It is clear that the python source has changed to read an extra long from the file before reading the code object. So my assumption is that we also need to read another long. Since we are supporting multiple python versions we will need some sort of "#if PY_MAJOR_VERSION..." to ensure we are compatible with both. It looks like the extra long was added in this change, which is python 3.3, so we should only read an extra long in versions 3.3 or newer.

@pjr1060 Can you try adding another PyMarshal_ReadLongFromFile in pyembed_run_pyc and see if that resolves the issue on python 3.5.3?

Member

bsteffensmeier commented May 10, 2017

I am not very familiar with this code but I think I might have it figured out. The comment on pyembed_run_pyc makes it clear that the code was gratuitously copyied from pythonrun.c::run_pyc_file. I compared the code in run_pyc_file in python 2.7 with the code in run_pyc_file in python 3.5. It is clear that the python source has changed to read an extra long from the file before reading the code object. So my assumption is that we also need to read another long. Since we are supporting multiple python versions we will need some sort of "#if PY_MAJOR_VERSION..." to ensure we are compatible with both. It looks like the extra long was added in this change, which is python 3.3, so we should only read an extra long in versions 3.3 or newer.

@pjr1060 Can you try adding another PyMarshal_ReadLongFromFile in pyembed_run_pyc and see if that resolves the issue on python 3.5.3?

@ndjensen

This comment has been minimized.

Show comment
Hide comment
@ndjensen

ndjensen May 19, 2017

Member

Fixed in dev_3.7.

Member

ndjensen commented May 19, 2017

Fixed in dev_3.7.

@pjr1060

This comment has been minimized.

Show comment
Hide comment
@pjr1060

pjr1060 May 20, 2017

I can confirm that this has fixed the issue on Python 3.5.3. Thank you!

pjr1060 commented May 20, 2017

I can confirm that this has fixed the issue on Python 3.5.3. Thank you!

@pjr1060

This comment has been minimized.

Show comment
Hide comment
@pjr1060

pjr1060 May 26, 2017

Hi Nathan - I just read your comment re: 3.7 release plan in issue #79

Since the release date of Jep 3.7 is somewhat undetermined at this time, I would like to fix this on the dev_3.6 branch and will make a new Jep 3.6 release soon.

If there is to be a 3.6 update would it be possible to apply this fix to 3.6 for general release? Thanks again for all your help!

pjr1060 commented May 26, 2017

Hi Nathan - I just read your comment re: 3.7 release plan in issue #79

Since the release date of Jep 3.7 is somewhat undetermined at this time, I would like to fix this on the dev_3.6 branch and will make a new Jep 3.6 release soon.

If there is to be a 3.6 update would it be possible to apply this fix to 3.6 for general release? Thanks again for all your help!

@ndjensen

This comment has been minimized.

Show comment
Hide comment
@ndjensen

ndjensen May 26, 2017

Member

Yeah, we should be able to do that. @bsteffensmeier, can you backport this fix to dev_3.6?

Member

ndjensen commented May 26, 2017

Yeah, we should be able to do that. @bsteffensmeier, can you backport this fix to dev_3.6?

@ndjensen ndjensen closed this in bd14a11 Jun 4, 2017

@ndjensen

This comment has been minimized.

Show comment
Hide comment
@ndjensen

ndjensen Jun 5, 2017

Member

This was backported into 3.6 and released as part of 3.6.4.

Member

ndjensen commented Jun 5, 2017

This was backported into 3.6 and released as part of 3.6.4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment