Unable to execute compiled script with JEP and Python 3.5.3

[pjr1060]

I have implemented a compile step to convert .py to .pyc files prior to execution. Each script is cached and if it is present it is used instead of the interpreted .py.

Everything works great when JEP is asked to execute the .pyc when on Python 2.7.13. However when I change environment and use Python 3.5.3 I get the following error...

Bad code object in .pyc file

Note I am aware of pyc magic numbers - the cached pyc files are cleaned and compiled with 3.5.3. Looking at pyembed.c I am getting past the magic number validation at line 1328 and seeing the error at the validation step at line 1334.

Here's the bit I can't understand. If I execute the compiled .pyc file using python3 directly it runs just fine.

I have attached a simple example that highlights the problem, the zip contains the following...

jepHello.py - native script
jepHello-2.7.13.pyc - compiled with 2.7.13 - runs from JEP on Python 2.7.13 and native python2
jepHello-3.5.3.pyc - compiled with 3.5.3 - fails with JEP on Python 3.5.3 but runs with native python3

jepHello.zip

Please let me know if there's something I'm missing or if this is a bug and needs more investigation I'd be very happy to help. JEP is an awesome piece of work!

Incidentally I am using JEP as the caching compiler too using the following ...

jep.eval("import py_compile");
String py="py_compile.compile(file='"+scriptFile.getAbsolutePath()+"', cfile='"+compiledFile.getAbsolutePath()+"' )";
jep.eval(py);
jep.eval(null); // flush

This works flawlessly and the JEP generated .pyc runs fine on any version of native python.

Thanks for any help you can offer...

Peter

[bsteffensmeier]

I am not very familiar with this code but I think I might have it figured out. The comment on pyembed_run_pyc makes it clear that the code was gratuitously copyied from pythonrun.c::run_pyc_file. I compared the code in run_pyc_file in python 2.7 with the code in run_pyc_file in python 3.5. It is clear that the python source has changed to read an extra long from the file before reading the code object. So my assumption is that we also need to read another long. Since we are supporting multiple python versions we will need some sort of "#if PY_MAJOR_VERSION..." to ensure we are compatible with both. It looks like the extra long was added in this change, which is python 3.3, so we should only read an extra long in versions 3.3 or newer.

@pjr1060 Can you try adding another PyMarshal_ReadLongFromFile in pyembed_run_pyc and see if that resolves the issue on python 3.5.3?

[ndjensen]

Fixed in dev_3.7.

[pjr1060]

I can confirm that this has fixed the issue on Python 3.5.3. Thank you!

[pjr1060]

Hi Nathan - I just read your comment re: 3.7 release plan in issue #79

Since the release date of Jep 3.7 is somewhat undetermined at this time, I would like to fix this on the dev_3.6 branch and will make a new Jep 3.6 release soon.

If there is to be a 3.6 update would it be possible to apply this fix to 3.6 for general release? Thanks again for all your help!

[ndjensen]

Yeah, we should be able to do that. @bsteffensmeier, can you backport this fix to dev_3.6?

[ndjensen]

This was backported into 3.6 and released as part of 3.6.4.