Custom builds of PyCall.jl

[benmoran]

It's currently possible to get pyjulia to work, at least on Linux, by using PyCall.jl at tag v1.3.0 together with this PR for PyJulia: #47

But a proper solution is still needed. @stevengj has some suggestions here:
JuliaPy/PyCall.jl#219 (comment)

In particular he suggests that each pyjulia ought to make a separate Pkg.clone of PyCall, and Pkg.build it specializing to the instance of Python that is being used.

[stevengj]

This should probably be done in conjunction with an PR to PyCall, so that if you set some environment variable during the Pkg.build phase then it looks up CPython symbols in the global namespace (for cases where libpython is statically linked into python). In this case, I'm hoping you can just use ccall(:Py_SomeSymbol, ...), which should be a trivial change to the @pysym macro in PyCall.

[stevengj]

I would also suggest cloning into a "private" directory for pyjulia, and push!(mydirectory, LOAD_PATH) so that Julia can find it. You can also push a private directory to Base.LOAD_CACHE_PATH so that the PyCall.ji cache file will be stored there. In particular, do something like:

using Compat
push!(mydirectory, LOAD_PATH)
push!(mydirectory, Base.LOAD_CACHE_PATH)
using PyCall
pop!(LOAD_PATH)
pop!(Base.LOAD_CACHE_PATH)

when you are initializing pyjulia. (Do using Compat first so that if it has been updated and needs to be recompiled, it's cache file will go into the standard Julia cache path rather than into your private cache path.)

[Keno]

The real problem here is that is python the precompile environment needs to look the same as the runtime environment. In particular that means that in the general case python needs to be in charge of the precompile environment as well. Talking with @vtjnash, we could create a python script that takes the appropriate arguments and pretends to be julia closely enough for julia to use it for precompilation.

[stevengj]

@Keno, I'm not sure I follow you. As long as we build with the right version of Julia, and use the pyjulia Pkg.build("PyCall") output, I don't see why precompilation within julia shouldn't work.

Oh, no, I see — the problem is that precompilation needs to run PyCall's __init__, and this will fail unless we are running inside Python.

[Keno]

Because that will try to load the symbols from the executable environment, which won't be there in the precompile process.

[Keno]

@vtjnash and I came up with something that I think is actually quite clever:

#!/usr/bin/env python
import ctypes
from ctypes import *
ctypes.CDLL("/Users/kfischer/Projects/julia-doublecheck/usr/lib/libjulia-debug.dylib", ctypes.RTLD_GLOBAL)
test = ctypes.CDLL("/Users/kfischer/Projects/julia-doublecheck/usr/bin/julia-debug", ctypes.RTLD_GLOBAL)
test.main.argtypes = [c_int,POINTER(c_char_p)]
test.main.restype = None
# Work around libuv needing argv strings to be adjacent in memory
argv = ["julia","--output-bc","test"]
argvlens = map(lambda str: len(str)+1, argv)
dataarr = (ctypes.c_char * sum(argvlens))()
argvarr = (ctypes.c_char_p * (len(argv)+1))()
idx = 0
i = 0
for arg in argv:
    dataarr[idx:(idx+len(arg))] = arg
    dataarr[idx+len(arg)] = '\0'
    argvarr[i] = ctypes.addressof(dataarr) + idx
    idx += len(arg) + 1
    i += 1
argc = c_int(len(argv))
test.main(argc, argvarr)

Seems to work ok. Will have to try across operating systems and fully end-to-end.

[Keno]

Ok, unfortunately this approach does not work on windows. Luckily, the Python FAQ states that On Windows, Python must be a DLL to handle importing modules that are themselves DLL’s. (This is the first key undocumented fact.) [1], so I think we can safely ignore that case.

[1] https://docs.python.org/2/faq/windows.html#how-can-i-embed-python-into-a-windows-application

[Keno]

Bad news on linux as well. Doing this corrupts the c runtime library state :(. I'll see if we can move the argument parsing into libjulia instead. That'll make this an 0.5 only thing but that may not be so bad.

[Keno]

Fixed by #54.