iPython Notebook integration #1330

Closed
StefanKarpinski opened this Issue Oct 3, 2012 · 30 comments

Comments

Projects
None yet
Owner

StefanKarpinski commented Oct 3, 2012

The web repl is not maintained and should be phased out in favor of a Julia backend to the iPython Notebook web frontend.

Member

fperez commented Oct 4, 2012

Awesome, pinging a few of the usual suspects so they're aware of this and we can participate in the discussion and help as needed... @ellisonbg, @minrk, @Carreau, @takluyver.

Carreau commented Oct 4, 2012

That would be great to know what component of the notebook are not configurable/reusable enough !
We'll do our best to adapt to your needs. Having 2 "natives" languages supported by IPython would be a great deal to show that we are open.

May I suggest a IPEP about Notebook polyglossia, to anyone who has time ? (support also .ijlnb,....i*nb file extension... etc...

Owner

StefanKarpinski commented Oct 4, 2012

Yes, I think this will be a fun and instructive ride. Do I recall correctly that there is also a Ruby backend? Or was that just a Ruby driver for the Python backend?

Owner

StefanKarpinski commented Oct 4, 2012

It's a good thing to look at nevertheless. Thanks for the link.

Does it also make sense to develop a %%julia cell magic? I recall someone started a discussion about integrating Julia with Python stuff. It would be really neat to be able to define a function in Julia in one cell, and call it in Python in the next cell - similar to our existing Cython and Bitey cell magics.

Also pinging @bfroehle, who wrote the bitey cell magic.

bfroehle commented Oct 4, 2012

Perhaps, but I don't know much about Julia. The Bitey magic was relatively easy to write because of the excellent bitey module by David Beazley.

Owner

StefanKarpinski commented Oct 4, 2012

@takluyver: that may be harder because the integration necessary to call Julia functions from Python isn't there yet. Once that level of interoperability is there, however, this is definitely an excellent idea.

I will be interested to see where this goes.

Member

fperez commented Oct 4, 2012

@takluyver, I think once a Julia/Python bridge matures, it will absolutely make sense to have %%julia, along with our %%R and %%octave ones, to cover all the key open source technical computing tools out of the box.

ivanov commented Mar 1, 2013

@fperez and @JeffBezanson have been working on a %%julia magic which you can find here.

Owner

JeffBezanson commented Mar 4, 2013

I have added https://github.com/JuliaLang/ipython, which now contains IPython/extensions/juliamagic.py. Here is how to mess with it:

  • Edit the paths to libjulia in juliamagic.py
  • In the ipython directory (the cloned repo), run python ipython.py
  • Type %load_ext juliamagic
  • Type %julia 1+2 to run something in julia
  • To run the web notebook, you'll need to install a few more dependencies, then run python ipython.py notebook

There are currently two implementations: JuliaMagics0 which works but is very limited, and JuliaMagics, which attempts to use @stevengj 's PyCall for full mapping of julia objects to python, which we don't have working yet.

Owner

stevengj commented Mar 4, 2013

Note that the PyCall JuliaMagics should be changed to use pyinitialize(C_NULL) for initialization -- we realized belatedly that you need to load symbols from the running Python instance rather than re-loading libpython.so. With this change, it correctly realizes that Python is initialized already: ccall(pyfunc(:Py_IsInitialized), Cint, ()) returns 1.

Unfortunately, it still segfaults in PyCall.pyimport when I try to ccall the PyImport_ImportModule function. I'm not sure what is going on.

Owner

stevengj commented Mar 4, 2013

Aha!, I figured it out! JuliaMagics needs to open Julia by calling ctypes.PyDLL, not ctypes.CDLL. (The latter releases Python's global interpreter lock, making it impossible to call many Python API functions.)

With this change, it looks like the PyObject translation now basically works:

In [1]: %load_ext juliamagic
>> J: using PyCall
t   : Nothing
>> J: pyinitialize(C_NULL)
t   : Nothing
>> J: PyObject
t   : CompositeKind

In [2]: %julia [1,2,3,4]
>> J: [1,2,3,4]
t   : Array
anstype: Array
pyo CompositeKind
xx type PyObject
Out[2]: array([1, 2, 3, 4])
Owner

JeffBezanson commented Mar 4, 2013

Excellent!

Member

fperez commented Mar 4, 2013

Sweet!! Many thanks, @stevengj for this. Will update the gist for the julia magic so that we can maintain it separately. Didn't make it in time for the MIT talk last Friday, but I'm giving one at Stanford on Friday and will have this to demo there.

Awesome job, guys!

Owner

stevengj commented Mar 4, 2013

Don't forget to incref the result of calling jpyobj, since otherwise you only have a borrowed reference. Not sure how to do this from Python... possibly pythonapi.Py_IncRef?

Owner

stevengj commented Mar 4, 2013

Also, note that we will be living dangerously in passing arrays from Julia to Python until I get the weak-reference stuff working to coordinate the Python gc with the Julia gc.

Owner

stevengj commented Mar 6, 2013

@fperez put up a git repository for juliamagics, and I've pushed a few changes to make the PyCall stuff more-or-less work (modulo some instability as mentioned above).

It now assumes you have a julia executable in your $PATH and uses it to find the location of the Julia library (rather than hardcoding the path). Probably it should just try loading libjulia-release.so from the system path first, and then only try running julia (which is a bit slow to launch) if this fails, or something.

Owner

stevengj commented Mar 6, 2013

Update: the latest PyCall now uses weakrefs to prevent Julia from garbage-collecting arrays until they are no longer needed by Python, so should be a bit safer to use now.

This is really exciting work!

Owner

stevengj commented Mar 10, 2013

I wonder if some of this stuff should go into a julia.py module for Python (not just IPython), so you can do something like the following in Python:

import Julia
fib = Julia.jl_eval('fib(n) = n < 2 ? n : fib(n-1) + fib(n-2)')
fib(20)

etcetera. (jl_eval and similar functions could initialize Julia, install the PyCall package, etcetera, the first time they are called.) Essentially all of the machinery for this is already in the IPython juliamagic.py file, it is just a matter of repackaging. Then the IPython magics file could just import Julia.

There is also the question of distribution. One option might be to distribute julia.py with Julia itself, and have Julia's make install automatically install it into the appropriate Python directory.

Member

pao commented Mar 10, 2013

Seems like it makes more sense on PyPI, where it can be a part of a pip requirements.txt file, even if it's developed in-tree with Julia.

Owner

stevengj commented Mar 10, 2013

(For my own part, I'd prefer to work mainly on the Julia and C side of the equation, and would prefer if someone else could take the lead with the glue on the Python side. As I mentioned to @fperez, I actually have written more code with the Python C API than I have in Python itself.)

A few points:

  • I like the idea of building the IPython machinery on top of a more general purpose julia model for python.
  • Any julia module for python should be a separate, standard python project with its own repo that can be installed like any other python project (registered on pypi, pip installable).
  • That project should probably include the IPython magics as well.
  • The two most logical places to host such a repo would be under the ipython or JuliaLang GitHub orgs. Not sure if it really makes a big different which it is...
Member

fperez commented Mar 11, 2013

On Sun, Mar 10, 2013 at 4:41 PM, Brian E. Granger
notifications@github.comwrote:

A few points:

  • I like the idea of building the IPython machinery on top of a more
    general purpose julia model for python.
  • Any julia module for python should be a separate, standard python
    project with its own repo that can be installed like any other python
    project (registered on pypi, pip installable).
  • That project should probably include the IPython magics as well.
  • The two most logical places to host such a repo would be under the
    ipython or JuliaLang GitHub orgs. Not sure if it really makes a big
    different which it is...

Yup, I was already planning on refactoring the current magic into a
'library' part and a magic piece, just as suggested. But I wasted a bunch
of time with a build issue (now resolved) so I fell a little behind.

And that pattern should apply in general: magics, because they require
special syntax to be used, should always be very thin wrappers around the
real brains of any system where all the logic is done by 'normal' python
code.

I actually would like to see the r/octave/cython ones refactored in that
form, so that their main objects are more easily used outside of the
special %% syntax.

Owner

stevengj commented Mar 15, 2013

Here is a great demo that @fperez posted to the mailing list: http://nbviewer.ipython.org/urls/raw.github.com/fperez/juliamagic/master/JuliaIPython.ipynb

Some refactoring and rearrangment still needs to be done to get full integration, e.g. to expose the Python Out[n] answers back in Julia as well as deeper integration, but the basic plumbing seems to be functional.

@vtjnash vtjnash closed this Mar 16, 2013

@JeffBezanson JeffBezanson reopened this Mar 16, 2013

Owner

staticfloat commented Apr 4, 2013

If anyone else is interested, the link above to the JuliaIPython notebook has been moved here, and development on this is happening in this repository. Great work, stevengj and fperez! This is some serious magic! (pun kinda-sorta intended)

Owner

ViralBShah commented Apr 28, 2013

I suggest we close this general issue, and track the ipython-julia integration with specific issues.

Member

fperez commented Apr 28, 2013

+1

@ViralBShah ViralBShah closed this Apr 28, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment