Permalink
Find file Copy path
8ff5047 Jan 2, 2019
1 contributor

Users who have contributed to this file

1211 lines (886 sloc) 38 KB

ob-ipython 2.0 in scimax

Updates to ipython in scimax

Scimax has used a forked version ob-ipython for a long time. The upstream version has improved a lot over the last year, and is nicer than my forked version, especially for asynchronous blocks. So…. I have been working to integrate the upstream version and replace my forked version. Well, mostly. I have specific things I want in org-mode/ipython integration that aren’t built in to ob-ipython, and aren’t customizable either. So, my new integration still monkey patches quite a few ob-ipython functions.

The new update is in two files: ./scimax-ob.el, which contains pretty general functions for all org babel src blocks I think, and ./scimax-org-babel-ipython-upstream.el which contains monkey-patches on ob-ipython, and additional features.

This document shows the new features.

I am anticipating deprecating the old ob-ipython fork, but I have no idea how many people use it, and have no issue with it, so I have added an ob-ipython-upstream submodule, and started a new scimax-org-babel-ipython-upstream.el library that is independent of the old one.

Basic output

Source blocks usually differentiate between results that are output and value. This has rarely made sense for how I use org-mode. I usually print things and want outputs. Anything you print will show in the results, and the last value will also show. Printed output comes first.

def f(x):
    print(x)
    return x**2

f(1.41)
1.9880999999999998

A new feature for scimax users is the execution count, which shows you the counter for when the cell was executed. This shows as comment, so it won’t appear in exported content. If you don’t like the count, you can turn it off with this in your init files. It defaults to on.

(setq ob-ipython-suppress-execution-count nil)

Ipython can have many kinds of output. The # text/plain line in the output tells you what kind of output this is. Later we will see how to filter these. If you don’t like these lines, turn them off like this.

(setq ob-ipython-show-mime-types nil)

Multiple inline figures and output

The output really shines with printed output and figures. In the upstream ob-ipython you cannot do both, and it only shows one figure. We get everything with scimax, including all the outputs. You can also filter the outputs to only show what you want.

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

t = np.linspace(0, 20 * np.pi, 350)
x = np.exp(-0.1 * t) * np.sin(t)
y = np.exp(-0.1 * t) * np.cos(t)

plt.plot(x, y)
plt.axis('equal')

plt.figure()
plt.plot(y, x)

plt.axis('equal')

print('Length of t = {}'.format(len(t)))
print('x .dot. y = {}'.format(x @ y))

obipy-resources/19a94859c854121de1c305426d09b77e-64787w6c.png

obipy-resources/19a94859c854121de1c305426d09b77e-647879Ej.png

Fine tune the output

By default we get all the outputs from the kernel. Here we get plain text, an image and a LaTeX representation.

from sympy import *
init_printing()
x, y, z = symbols('x y z')

Integral(sqrt(1 / x), x)

obipy-resources/19a94859c854121de1c305426d09b77e-64787KPp.png

You can choose to display in the src block header by specifying which mime-type to display (that is one reason I added it to the output).

Integral(sqrt(1/x), x)

I have not added an exclude feature, but it would not be hard to. Unfortunately, it is not yet possible to control the order these come out in, or to do things like add captions or size information to the images.

Exceptions are better

By default we capture exceptions in results. I like that because for notes, it is nice to show what happens, and I find it less disruptive while working to not have windows opening and closing.

print(1 / 0)

ZeroDivisionErrorTraceback (most recent call last) <ipython-input-7-3ec96714f820> in <module>() —-> 1 print(1 / 0)

ZeroDivisionError: division by zero

If you like an exception buffer, set this variable like this:

(setq ob-ipython-exception-results nil)

Even this is better, press q in that buffer to jump to the offending line in your src block. This does not work right when you run in async mode.

print(1 / 0)

Documentation and code via ? and ??

You can now access the documentation like you can in jupyter.

import numpy as np
?np.linspace

Similarly you can see the source code:

??np.linspace

This works for functions defined in your org file too.

def func(X):
    '''a function of X'''

    return 2 * X
?func

Inspecting objects in src-blocks

You can inspect things in your src-blocks to find out things about them. With your cursor on some object, type s-/ and if it can be figured out you will get some information about it.

import numpy as np

x = np.linspace(0, 1, 5)
x

You can also enable something like eldoc with elisp:scimax-ob-ipython-turn-on-eldoc and turn it off with elisp:scimax-ob-ipython-turn-off-eldoc. This is a little tricky, it sometimes only works when you have a syntactically correct command with parentheses, or after some cursor movement. I wish it worked better.

np.linspace

Completion in src-blocks

You can get some completion options. I like scimax-ob-ipython-complete-ivy. I have this bound to s-. It often works on objects too.

np.linalg.add
x.argsort

You can also use company mode like this:

(add-to-list 'company-backends 'company-ob-ipython)
(company-mode)
np.linalg.inv

Company-mode is kind of slow, and lacks the completion like ivy, but it might work for you.

Easy async

You can use an :async header to run a block asynchronously. That means it runs in the background and you can keep using emacs! You can mix and match async blocks in a document. I simplified how this is done compared to upstream; in my version just putting :async in the header (with no argument) makes it run asynchronously.

import time

for i in range(4):
    print(i)
    time.sleep(2)
# type new things
# keep on working!
print('done')

You will see another buffer pop up with intermediate results, and they will be put back in the results when it is done.

Customizing outputs

ipython/org-mode really shines when you start leveraging rich outputs from Ipython. A new feature I have added is that you can write your own functions to customize the output.

This variable maps mime-types to formatting functions. You can add new mime-types to this, or redefine the formatting functions if you don’t like the way the work.

(append '(("mime-type" "formatting function"))
	'(hline)
	(loop for (mime-type . func) in ob-ipython-mime-formatters
	      collect (list mime-type func)))
mime-typeformatting function
text/plainob-ipython-format-text/plain
text/htmlob-ipython-format-text/html
text/latexob-ipython-format-text/latex
text/orgob-ipython-format-text/org
image/pngob-ipython-format-image/png
image/svg+xmlob-ipython-format-image/svg+xml
application/javascriptob-ipython-format-application/javascript
defaultob-ipython-format-default
outputob-ipython-format-output

You can set these to whatever you want, and add new ones for new mimetypes.

Better representations of Polynomial objects

Most python objects have a __str__ or __repr__ method defined that display them when printed. For example, here is a Polynomial from numpy with it’s default representation.

import numpy as np
p = np.polynomial.Polynomial([1, 2, 3])
p

Let’s change this to get a LaTeX representation (adapted from https://github.com/jupyter/ngcm-tutorial/blob/master/Part-1/IPython%20Kernel/Custom%20Display%20Logic.ipynb). We will do this on the Ipython side of output customization where we register a formatting function for a specific type in IPython.

def poly_to_latex(p):
    terms = ['%.2g' % p.coef[0]]
    if len(p) > 1:
        term = 'x'
        c = p.coef[1]
        if c != 1:
            term = ('%.2g ' % c) + term
        terms.append(term)
    if len(p) > 2:
        for i in range(2, len(p)):
            term = 'x^%d' % i
            c = p.coef[i]
            if c != 1:
                term = ('%.2g ' % c) + term
            terms.append(term)
    px = '$P(x)=%s$' % '+'.join(terms)
    dom = r', $x \in [%.2g,\ %.2g]$' % tuple(p.domain)
    return px + dom


ip = get_ipython()
latex_f = ip.display_formatter.formatters['text/latex']
latex_f.for_type_by_name('numpy.polynomial.polynomial',
                                 'Polynomial', poly_to_latex)
p

That looks nice, but we can go one step further and define graphical outputs too.

import matplotlib.pyplot as plt
from IPython.core.pylabtools import print_figure

def poly_to_png(p):
    fig, ax = plt.subplots()
    x = np.linspace(-1, 1)
    y = [p(_x) for _x in x]
    ax.plot(x, y)
    ax.set_title(poly_to_latex(p))
    ax.set_xlabel('x')
    ax.set_ylabel('P(x)')
    data = print_figure(fig, 'png')
    # We MUST close the figure, otherwise IPython's display machinery
    # will pick it up and send it as output, resulting in a double display
    plt.close(fig)
    return data


ip = get_ipython()
png_f = ip.display_formatter.formatters['image/png']
png_f.for_type_by_name('numpy.polynomial.polynomial',
                                 'Polynomial', poly_to_png)

Now, we can easily see the formula and shape of this polynomial in a graphical form.

p

Most likely you would not put all this code into a document like this, but would instead put it in a Python library you import. The point here is to show what can be done with that, and once it is done, you get easy visualization of objects.

Tensorflow visualizations

In Tensorflow, we are always making computation graphs. These are usually visualized in Tensorboard. We can leverage Jupyter to show us a graphical representation instead. This is another example of registering a type in Ipython.

from graphviz import Digraph

def tf_to_dot(graph):
    "Adapted from https://blog.jakuba.net/2017/05/30/tensorflow-visualization.html"
    dot = Digraph()

    for n in g.as_graph_def().node:
        dot.node(n.name, label=n.name)

        for i in n.input:
            dot.edge(i, n.name)
    dot.format = 'svg'
    return dot.pipe().decode('utf-8')

ip = get_ipython()
svg_f = ip.display_formatter.formatters['image/svg+xml']
svg_f.for_type_by_name('tensorflow.python.framework.ops',
                       'Graph', tf_to_dot)
import tensorflow as tf

g = tf.Graph()

with g.as_default():
    a = tf.placeholder(tf.float32, name="a")
    b = tf.placeholder(tf.float32, name="b")
    c = a + b

g

obipy-resources/19a94859c854121de1c305426d09b77e-64787j3K.svg

Now we have a record of what the graph looks like. Ez peezy.

Pandas in org-mode

Just for fun, here is a way to get Pandas dataframes to be displayed as org-mode tables using tabulate (https://pypi.python.org/pypi/tabulate). This is adapted from https://github.com/gregsexton/ob-ipython. tabulate has a built-in org formatter, so no reason to reinvent that!

import IPython
import tabulate

class OrgFormatter(IPython.core.formatters.BaseFormatter):
    format_type = IPython.core.formatters.Unicode('text/org')
    print_method = IPython.core.formatters.ObjectName('_repr_org_')

def pd_dataframe_to_org(df):
    return tabulate.tabulate(df, headers='keys', tablefmt='orgtbl', showindex='always')

ip = get_ipython()
ip.display_formatter.formatters['text/org'] = OrgFormatter()

f = ip.display_formatter.formatters['text/org']
f.for_type_by_name('pandas.core.frame', 'DataFrame', pd_dataframe_to_org)

import pandas as pd
df = pd.DataFrame([1, 2], columns=['widecolumn'])
df.index.name = 'indexname'
df
indexnamewidecolumn
01
12

Here is a bigger example.

import numpy as np
df2 = pd.DataFrame(np.random.randint(low=0, high=10, size=(5, 5)),
                   columns=['a', 'b', 'c', 'd', 'e'])
df2
abcde
062439
161398
279700
362239
431998

Customizing a class output with repr_* methods

Adapted from http://nbviewer.jupyter.org/github/ipython/ipython/blob/6.x/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb#Custom-Mimetypes-with-_repr_mimebundle

The canonical way to make rich outputs on your own classes is to add repr_X methods. Here is the example from the Jupyter tutorial listed above. Here, we just add a PNG and LaTeX representation.

import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt

from IPython.core.pylabtools import print_figure
from IPython.display import Image, SVG, Math

class Gaussian(object):
    """A simple object holding data sampled from a Gaussian distribution.
    """
    def __init__(self, mean=0.0, std=1, size=1000):
        self.data = np.random.normal(mean, std, size)
        self.mean = mean
        self.std = std
        self.size = size
        # For caching plots that may be expensive to compute
        self._png_data = None

    def _figure_data(self, format):
        fig, ax = plt.subplots()
        ax.hist(self.data, bins=50)
        ax.set_title(self._repr_latex_())
        ax.set_xlim(-10.0,10.0)
        data = print_figure(fig, format)
        # We MUST close the figure, otherwise IPython's display machinery
        # will pick it up and send it as output, resulting in a double display
        plt.close(fig)
        return data

    def _repr_png_(self):
        if self._png_data is None:
            self._png_data = self._figure_data('png')
        return self._png_data

    def _repr_latex_(self):
        return r'$\mathcal{N}(\mu=%.2g, \sigma=%.2g),\ N=%d$' % (self.mean,
                                                                 self.std, self.size)
Gaussian()

obipy-resources/19a94859c854121de1c305426d09b77e-64787wBR.png

text/org output with repr_mimebundle

We can define custom outputs for our own objects too. Here we define org and html representations of a heading object within the class. We have to define a repr_mimebundle method to get ‘text/org’ output as it is not a predefined type in Jupyter. Alternatively, we could use the methods earlier to define formatters for these types.

See http://nbviewer.jupyter.org/github/ipython/ipython/blob/6.x/examples/IPython%20Kernel/Custom%20Display%20Logic.ipynb#Custom-Mimetypes-with-_repr_mimebundle_ for more details.

class Heading(object):
    def __init__(self, content, level=1, tags=()):
        self.content = content
        self.level = level
        self.tags = tags

    def _repr_org(self):
        s = '*' * self.level + ' ' + self.content
        if self.tags:
            s += f"  :{':'.join(self.tags)}:"
        return s

    def _repr_html(self):
        return  f"<h{self.level}>{self.content}</h{self.level}>"

    def _repr_mimebundle_(self, include, exclude, **kwargs):
        """
        repr_mimebundle should accept include, exclude and **kwargs
        """

        data = {'text/html': self._repr_html(),
                'text/org': self._repr_org()
                }
        if include:
            data = {k:v for (k,v) in data.items() if k in include}
        if exclude:
            data = {k:v for (k,v) in data.items() if k not in exclude}
        return data

Now, you can construct headings in iPython, and get different outputs that might be suitable for different purposes.

Heading('A level 4 headline', level=4, tags=['example'])

Bokeh

The Jupyter notebook does really shine for JavaScript driven interactive data exploration. For now, the only option for Emacs is to open external programs for this, e.g. a matplotlib figure, or a browser. Bokeh is a really interesting interactive plotting library you can use in Python, but it makes interactive html documents for viewing in a browser. Here we will adapt the outputs to show us a thumbnail and org-link to open the html file. This is yet another example of registering a type in Ipython.

Here we modify the plain text output so that it saves an html file, and returns a link to it.

Note you need to install bokeh, selenium, pillow with conda, and install a modern phantomjs in your OS for this to work (I build one from https://github.com/eisnerd/phantomjs).

import tempfile
import warnings
warnings.filterwarnings("ignore")
import webbrowser

import IPython
from bokeh.io import export_png
from bokeh.io.saving import save
from bokeh.plotting.figure import Figure
import os

def bokeh_to_org(plt):
  fh, tmp = tempfile.mkstemp(suffix=".html", prefix="bokeh-",
                             dir="obipy-resources")
  fname = save(plt, tmp)
  os.close(fh)
  os.system(f'open {fname}')
  return "[[{}]]".format(fname)


def bokeh_to_png(plt):
  png_filename = export_png(plt)
  with open(png_filename, "rb") as f:
    return f.read()


def bokeh_mimebundle(self, include=(), exclude=(), **kwargs):
  data = {'text/org': bokeh_to_org(self),
          'image/png': bokeh_to_png(self)}

  if include:
    data = {k:v for (k,v) in data.items() if k in include}
  if exclude:
    data = {k:v for (k,v) in data.items() if k not in exclude}
  return data, {}

Figure._repr_mimebundle_ = bokeh_mimebundle

Now we are setup to make an interactive figure.

from bokeh.io import output_file, show
from bokeh.models import ColumnDataSource, HoverTool

from bokeh.sampledata.periodic_table import elements
from bokeh.transform import dodge, factor_cmap

periods = ["I", "II", "III", "IV", "V", "VI", "VII"]
groups = [str(x) for x in range(1, 19)]

df = elements.copy()
df["atomic mass"] = df["atomic mass"].astype(str)
df["group"] = df["group"].astype(str)
df["period"] = [periods[x - 1] for x in df.period]
df = df[df.group != "-"]
df = df[df.symbol != "Lr"]
df = df[df.symbol != "Lu"]

cmap = {
    "alkali metal": "#a6cee3",
    "alkaline earth metal": "#1f78b4",
    "metal": "#d93b43",
    "halogen": "#999d9a",
    "metalloid": "#e08d49",
    "noble gas": "#eaeaea",
    "nonmetal": "#f1d4Af",
    "transition metal": "#599d7A",
}

source = ColumnDataSource(df)

p = Figure(
    title="Periodic Table (omitting LA and AC Series)",
    plot_width=1000,
    plot_height=450,
    tools="",
    toolbar_location=None,
    x_range=groups,
    y_range=list(reversed(periods)))

p.rect(
    "group",
    "period",
    0.95,
    0.95,
    source=source,
    fill_alpha=0.6,
    legend="metal",
    color=factor_cmap(
        "metal", palette=list(cmap.values()), factors=list(cmap.keys())))

text_props = {"source": source, "text_align": "left", "text_baseline": "middle"}

x = dodge("group", -0.4, range=p.x_range)

r = p.text(x=x, y="period", text="symbol", **text_props)
r.glyph.text_font_style = "bold"

r = p.text(
    x=x,
    y=dodge("period", 0.3, range=p.y_range),
    text="atomic number",
    **text_props)
r.glyph.text_font_size = "8pt"

r = p.text(
    x=x, y=dodge("period", -0.35, range=p.y_range), text="name", **text_props)
r.glyph.text_font_size = "5pt"

r = p.text(
    x=x,
    y=dodge("period", -0.2, range=p.y_range),
    text="atomic mass",
    **text_props)
r.glyph.text_font_size = "5pt"

p.text(
    x=["3", "3"],
    y=["VI", "VII"],
    text=["LA", "AC"],
    text_align="center",
    text_baseline="middle")

p.add_tools(
    HoverTool(tooltips=[
        ("Name", "@name"),
        ("Atomic number", "@{atomic number}"),
        ("Atomic mass", "@{atomic mass}"),
        ("Type", "@metal"),
        ("CPK color", "$color[hex, swatch]:CPK"),
        ("Electronic configuration", "@{electronic configuration}"),
    ]))

p.outline_line_color = None
p.grid.grid_line_color = None
p.axis.axis_line_color = None
p.axis.major_tick_line_color = None
p.axis.major_label_standoff = 0
p.legend.orientation = "horizontal"
p.legend.location = "top_center"

p

obipy-resources/19a94859c854121de1c305426d09b77e-647879LX.png

Now if you click on the link above, it will open an interactive html file in your browser. It is just a tempfile, so some work might be necessary to get it to a persistent place, like the images are.

More complex display with ipython_display

This example is also from this url, and shows how to override the IPython output all together.

import numpy as np
import json
import uuid
from IPython.display import display_javascript, display_html, display

class FlotPlot(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.uuid = str(uuid.uuid4())

    def _ipython_display_(self):
        json_data = json.dumps(list(zip(self.x, self.y)))
        display_html('<div id="{}" style="height: 300px; width:80%;"></div>'.format(self.uuid),
            raw=True)

        display_javascript(f'''require(["//cdnjs.cloudflare.com/ajax/libs/flot/0.8.2/jquery.flot.min.js"], function() {{
          var line = JSON.parse("{json_data}");
          console.log(line);
          $.plot("#{self.uuid}", [line]);
        }});''', raw=True)

x = np.linspace(0,10)
y = np.sin(x)
FlotPlot(x, np.sin(x))
require(["//cdnjs.cloudflare.com/ajax/libs/flot/0.8.2/jquery.flot.min.js"], function() {
          var line = JSON.parse("[[0.0, 0.0], [0.20408163265306123, 0.20266793654820095], [0.40816326530612246, 0.39692414892492234], [0.6122448979591837, 0.5747060412161791], [0.8163265306122449, 0.7286347834693503], [1.0204081632653061, 0.8523215697196184], [1.2244897959183674, 0.9406327851124867], [1.4285714285714286, 0.9899030763721239], [1.6326530612244898, 0.9980874821347183], [1.836734693877551, 0.9648463089837632], [2.0408163265306123, 0.8915592304110037], [2.2448979591836737, 0.7812680235262639], [2.4489795918367347, 0.6385503202266021], [2.6530612244897958, 0.469329612777201], [2.857142857142857, 0.28062939951435684], [3.0612244897959187, 0.0802816748428135], [3.2653061224489797, -0.12339813736217871], [3.4693877551020407, -0.3219563150726187], [3.673469387755102, -0.5071517094845144], [3.8775510204081636, -0.6712977935519321], [4.081632653061225, -0.8075816909683364], [4.285714285714286, -0.9103469443107828], [4.4897959183673475, -0.9753282860670456], [4.6938775510204085, -0.9998286683840896], [4.8979591836734695, -0.9828312039256306], [5.1020408163265305, -0.9250413717382029], [5.3061224489795915, -0.8288577363730427], [5.510204081632653, -0.6982723955653996], [5.714285714285714, -0.5387052883861563], [5.918367346938775, -0.35677924089893803], [6.122448979591837, -0.16004508604325057], [6.326530612244898, 0.04333173336868346], [6.530612244897959, 0.2449100710119793], [6.73469387755102, 0.4363234264718193], [6.938775510204081, 0.6096271964908323], [7.142857142857143, 0.7576284153927202], [7.346938775510204, 0.8741842988197335], [7.551020408163265, 0.9544571997387519], [7.755102040816327, 0.9951153947776636], [7.959183673469388, 0.9944713672636168], [8.16326530612245, 0.9525518475314604], [8.36734693877551, 0.8710967034823207], [8.571428571428571, 0.7534867274396376], [8.775510204081632, 0.6046033165061543], [8.979591836734695, 0.43062587038273736], [9.183673469387756, 0.23877531564403087], [9.387755102040817, 0.03701440148506237], [9.591836734693878, -0.1662827938487564], [9.795918367346939, -0.3626784288265488], [10.0, -0.5440211108893699]]");
          console.log(line);
          $.plot("#2ebc40f4-a6da-4ae4-b45f-78288681d551", [line]);
        });

Clickable buttons in src blocks

Jupyter has some nice features like buttons to click on to run a block. We have something like that. The text in angle brackets in the comments below is clickable!

# <run>  <restart and run>  <repl>  <delete block>  <menu>
# open some buffers  <output>  <execute>  <debug>
6 * 3
5

Jupyter-like keybindings in src-blocks

Jupyter notebooks have some nice key bindings, like all the variations of modified-return that do different things. When your cursor is in an ipython block, these bindings are active. They are not active outside of ipython code blocks. See this magical post for how that is possible!

Ctrl-<return>run current block
Shift-<return>run current block and go to next one, create one if needed
Meta-<return>runs the current cell and inserts a new one below.
super-<return>restart ipython and run block
Meta-super-<return>restart ipython and run all blocks to point
H-<return>restart ipython and run all blocks in buffer
5

1

2
3

Note you can put :restart in the src block header and ipython will restart every time you run that block. This is helpful when developing libraries, as it forces the library to be reloaded every time you run the block.

H-=insert src-block above current block
C-u hyper-=insert src-block below current block
H–split current block at point, point in upper block
C-u H–split current block at point, point in lower block
H-hEdit the src block header in the minibuffer
H-wKill the current block
H-nCopy the current block
H-oClone the current block (make a copy of it below the current one
H-mMerge blocks in the selected region
s-wMove current block before the previous one
s-sMove current block below the next one
H-lClear results from the block
H-s-lClear all results in buffer
s-iJump to previous block
s-kJump to next block
H-qJump to a visible block with avy
H-s-qJump to a block in the buffer with ivy
s-/get help about thing at point (ob-ipython-inspect)
H-rswitch to session REPL
H-kKill the kernel

I am not 100% committed to all these bindings. I still find the need to fine tune them every once in a while.

Can’t remember all these bindings? Me neither. Checkout M-x scimax-obi/body (usually bound to H-s) for a nice hydra. The hydra key-bindings don’t match the ones in the tables above; I am not sure that makes sense. It would add keystrokes, but it would also be a good reminder of the bindings.

These keybindings are relatively easy to customize. The are stored as cons cells in this variable.

ob-ipython-key-bindings

You can define/change any binding you want like this. They are only active in ipython src blocks. For example, you can define a src-block key like this:

(scimax-define-src-key ipython "H-/" #'some-command)

If you like the Jupyter keybindings more directly, checkout these bindings when you are in an ipython block:

H-efunctionscimax-jupyter-edit-mode/body
H-cfunctionscimax-jupyter-command-mode/body

Other languages

Jupyter can run many languages ranging from Fortran to shell. I like hylang, a lispy Python. Install the hylang Jupyter kernel like this:

pip install git+https://github.com/Calysto/calysto_hy.git --user
python -m calysto_hy install --user

Now we can use it in scimax. This is already pre-configured in scimax. For fun, we use hylang here.

(import [tensorflow :as tf])

(setv a (tf.constant 3)
      b (tf.constant 3)
      c (tf.add a b))

(with [sess (tf.Session)] (print (.run sess c)))

There is a little issue with the double colon on output, but otherwise it looks like it works.

This kernel is a little quieter on exceptions than I would like, but still it could be useful if you want to play around with hylang and document your work.

Unique kernel per org file is default

By default, each org file gets a unique kernel. I am sure there is a use case for every buffer sharing one kernel, but it is too confusing for me, and too prone to errors where one buffer changes a variable that affects others. So, if you want src blocks in different buffers to share kernels, you have to manually specify the kernel in the header, or use this for the original behavior.

(setq ob-ipython-buffer-unique-kernel nil)

Multiple sessions in one org file

See this comment for an example of multiple remote sessions, and Remote kernels about remote kernels. Here we use a properties drawer in two different headlines to have different sessions open in the same org file. Of course, you can manually define the :session in src blocks, but using a header like this:

:PROPERTIES:
:header-args:ipython: :session session-1
:END:

means that you don’t have to type that on every block. That is helpful, since if you forget the block will use the default buffer kernel. This shows I am currently running four kernels. See this comment for an example of multiple remote sessions.

(ob-ipython--get-kernel-processes)

Session 1

Every block under this header will use the kernel associated with session-1. Here we just look at what ports are being used.

%connect_info

Session 2

Every block in this heading will use the kernel associated with session-2. You can see here it uses different ports than session-1 does. They are both running at the same time though.

%connect_info

Remote kernels

According to a comment in issue 114 this finally works. I don’t have a remote server to test it on, but I did test it locally. Here are the steps to follow, adapted from https://vxlabs.com/2017/11/30/run-code-on-remote-ipython-kernels-with-emacs-and-orgmode/.

Note: This comment reports some flakiness with the method below, and suggests an alternative approach that they find more robust.

On the remote server, find out where your runtime directory is like this:

jupyter --runtime-dir

Start a kernel on the remote server like this. It will create a json file that you will need to copy to your local machine jupyter runtime directory.

ipython kernel

You should see some text like this one.

NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049


To connect another client to this kernel, use:
    --existing kernel-16577.json

You can see the connection file is indeed in the runtime directory:

ls `jupyter --runtime-dir`

On a real remote machine, you would have to copy this file to your local machine runtime directory. There is nothing too mysterious in this file, it just has some information about the IP address and ports used.

cat `jupyter --runtime-dir`/kernel-226309.json

Now, on your local machine, connect to the remote kernel like this, obviously with your own username and hostname.

jupyter console --existing kernel-226309.json --ssh username@hostname

You should get some output like:

[ZMQTerminalIPythonApp] Forwarding connections to 127.0.0.1 via kitchin@some.host.com
[ZMQTerminalIPythonApp] To connect another client via this tunnel, use:
[ZMQTerminalIPythonApp] --existing kernel-16577-ssh.json
Jupyter console 5.2.0

Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

The critical text to note here is --existing kernel-16577-ssh.json. You need to specify that kernel file (note the -ssh in it) as the :session in the header of the src block.

The command above opens a local Ipython terminal. In that terminal, I typed a = 7.

Then, you specify the kernel connection file as the :session argument in an ipython block.

print(a)

b = 6

You can see from the output that we successfully connected to the kernel and that indeed a=7. Furthermore, in the block above we assigned b=6, and in the terminal I can type “b” and see that it is equal to 6.

If you have a lot of blocks in your buffer, you can use a file property like this to specify the header for every block in the buffer. See Multiple sessions in one org file for an example of using heading properties instead.

#+PROPERTY: header-args:ipython :session kernel-16577-ssh.json

Known issues

Interrupting the kernel does not work

This seems to be an issue with the upstream repo. It may work on unix/Mac, but maybe not on Windows. Probably does not work on remote kernels. See https://github.com/gregsexton/ob-ipython/commit/fcb2c48f64ba1005ad6fe5d1a4149f1117a9b45d