pyc.py: generates exes which fail on using std python libs

[slide]

From @ironpythonbot on December 9, 2014 17:59

I tried to generate a standalone exe for a tool I have (3 modules) but it
bombs out at startup with:

Unhandled Exception: IronPython.Runtime.Exceptions.ImportException: No module
named os  
at DLRCachedCode.__main__$1(CodeContext $globalContext, FunctionCode
$functionCode)  
at IronPython.Compiler.OnDiskScriptCode.Run()  
at IronPython.Runtime.PythonContext.InitializeModule(String fileName,
ModuleContext moduleContext, ScriptCode scriptCode, ModuleOptions options)  
at IronPython.Runtime.ModuleLoader.load_module(CodeContext context, String
fullName)  
at Microsoft.Scripting.Interpreter.FuncCallInstruction`4.Run(InterpretedFrame
frame)  
at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)  
at Microsoft.Scripting.Interpreter.LightLambda.Run4[T0,T1,T2,T3,TRet](T0 arg0,
T1 arg1, T2 arg2, T3 arg3)  

I can reproduce with this simple foo.py module

#  foo.py

import os  
print "Hello"  
print "OS:",os.name  
print "Bye"  

#  end foo.py

which I compile with the following:

ipy -m pyc /standalone /main:foo.py /target:exe /out:foo

or

ipy -m pyc /main:foo.py /target:exe /out:foo2

Starting either foo.exe or foo2.exe fails with above error.

If I remove the import os and the line that uses on.name, I get an executable
that works...

Work Item Details

Original CodePlex Issue: Issue 32420
Status: Proposed
Reason Closed: Unassigned
Assigned to: slide_o_mix
Reported on: Mar 15, 2012 at 8:48 PM
Reported by: ddewaleffe
Updated on: Feb 22, 2013 at 2:09 AM
Updated by: ddewaleffe

Plaintext Attachments

CodePlex Issue #32420 Plain Text Attachments

Binary Attachments

pyc-ddewaleffe.zippyc-ddewaleffe-v2.zip

Copied from original issue: IronLanguages/main#997

[slide]

From @ironpythonbot on December 9, 2014 17:59

On 2012-03-16 04:38:00 UTC, slide_o_mix commented:

What pyc.py needs to do is try and find all the modules that are needed and automatically compile them as well.

[slide]

From @ironpythonbot on December 9, 2014 17:59

On 2012-03-16 21:23:10 UTC, ddewaleffe commented:

I started to look at this last night... Hope to get this done over the week-end.

More news soon...

[slide]

From @ironpythonbot on December 9, 2014 17:59

On 2012-03-16 22:01:36 UTC, slide_o_mix commented:

ModuleFinder won't work for IronPython as it stands now, so there may need to be another avenue found.

[slide]

From @ironpythonbot on December 9, 2014 17:59

On 2012-03-18 00:10:46 UTC, ddewaleffe commented:

I attach findeps.py and my modified pyc.py (packaged in zip)

This relies on ast.py to recursively find imports and generate a list of files to be compiled along the main ones.

It works for the small examples I tried.
It may miss dependencies if ast.py fails to parse a source file (e.g: subprocess.py, random.py)

[slide]

From @ironpythonbot on December 9, 2014 17:59

On 2012-03-18 01:35:43 UTC, slide_o_mix commented:

What license would you attribute to finddeps.py? I think we would need it something similar to Apache in order to include it.

[slide]

From @ironpythonbot on December 9, 2014 17:59

On 2012-03-19 05:44:28 UTC, ddewaleffe commented:

Here is a new version of finddeps.py:

includes mention of licensing (apache v2) falls back of re based parsing (imperfect) for the cases where ast fails.... (see secondBestGuess() function)

[slide]

From @ironpythonbot on December 9, 2014 18:0

On 2012-03-19 15:21:57 UTC, ddewaleffe commented:

Fixed bug in latest finddeps.py. Wrong type returned...

[slide]

From @ironpythonbot on December 9, 2014 18:0

On 2012-03-19 15:27:03 UTC, ddewaleffe commented:

However, still does not work completely. It fails for complex modules like 'logging' implemented in a directory... More to do for tonight...

[slide]

From @ironpythonbot on December 9, 2014 18:0

On 2012-03-28 03:05:03 UTC, ddewaleffe commented:

Got a bit further. This latest package is able to get at the packages
init.py files too and tries m.n....mm.nn.py as well as m.n....mm.py or similarly for
init.py
So I get many more of the needed module...

However, if I generate pyc.exe using ipy pyc.py .....
the generated executable seems to contain the needed modules but fails to work as expected. Further I get no errors.... So at this stage I am a bit lost....

I am not claiming this code is the cleanest....

On the samples I tried, it seems that one quickly pulls in quite a lot of the std lib... Making me wonder if, indeed packaging the whole of the stdlib in zip file would not just be right...

But is it possible to package this also in the generated executable.

My hope is really to get to a single .exe that contains all that's needed....

[slide]

From @ironpythonbot on December 9, 2014 18:0

On 2012-03-29 06:49:47 UTC, slide_o_mix commented:

So you think it would be good to have a way of embedding the stdlib as a zip into the executable instead of compiling it all up?

[slide]

From @ironpythonbot on December 9, 2014 18:0

On 2012-03-29 12:27:59 UTC, ddewaleffe commented:

My primary concern is to be able to produce a real standalone exe. 1 file to copy somewhere. No requirement to copy/install anything else. Dont care if big...

This first trial I made sort of partly work. For instance it works on:
import os.path
print "Hello"
print os.path.realpath('.')
print "running on", os.name
print "Bye"

But it does not (no error message at all) for pyc.py. Don't know why at the moment. It surely worth looking at why it fails....

Looking at the list of dependencies pulled in in each case, they are many.... so why not include the whole of stdlib at once instead of trying to analyze it by parsing at every compilation. This compiled version could be cached to avoid doing it every time...

Now we have 2 ways to do this:
a. package the zip file and rely on zimpimport doing its work
b. compile all the files of the library and package them all in the executable

[slide]

Related to IronLanguages/main#991