Jupyter #399

scopatz · 2015-09-30T05:35:27Z

Added hooks for Jupyter notebooks to be automatically installed. Now the only thing is that this won't come with the pip or conda packages unless we make the packages dependent on jupyter / ipython. Does this sound reasonable to people?

melund · 2015-09-30T09:28:34Z

It does. If people install Jupiter later they just have to reinstall xonsh.

This is really great. I have some unfinished attempts to implement this as well. I am glad you found the time to do it the right way. I will try it out as soon as possible :)

melund · 2015-09-30T13:36:47Z

Ok. The first problem I encountered was that the kernel spec was not installed correctly when building conda packages. I have created PR to your jupyter branch that fixes this.

I haven't got the kernel to work though. It seems to start correctly even though I get this error

[I 15:34:37.924 NotebookApp] Creating new notebook in
[I 15:34:39.565 NotebookApp] Kernel started: dcf75d0c-58e6-4aea-935d-5631543d050d
[IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open'

But the notebook does't show any output when entering commands.

scopatz · 2015-09-30T16:23:24Z

Hi @melund - Awesome :)

As per the issue that you are having, have you installed jupyter_client and/or are you on IPython v4.0+? This is what I am targeting...

melund · 2015-09-30T21:09:57Z

I am using IPython 4.0, and jupyter_client is installed.. I will test again tomorrow.

asmeurer · 2015-10-01T01:18:23Z

setup.py

+            "codemirror_mode":"shell",
+            }
+    if CONDA:
+        d = os.path.join(os.environ['PREFIX'] +


This should be the same as sys.prefix

Yes. You are right. It has been a while since I tingled with these things. It is of course python in the build environment that runs setup.py, so it should be the same sys.prefix, when building.

I am not too happy about this being in the setup.py file, but I haven't seen any good ways of installing the kernel_spec with conda. You did something for the r-kernel, right? I think I saw it on conda-recipes, but I can't seem to find it now. Maybe it should be a post-install script that tests if Jupyter is there, and then runs the install_kernel_spec() function.

asmeurer · 2015-10-01T01:19:53Z

I don't think conda should have a monopoly on installing things to the install prefix, but maybe that's a more a gripe for jupyter team than xonsh.

melund · 2015-10-01T06:59:59Z

It would be nice if conda could label the kernel_spec file with the environment when installing. Then we would be able to see which environment the kernel belonged to directly from the notebook.

Let's say that xonsh was installed in three environments; root, dev and test. In the notebook, it only shows one entry, of the one install last. If the conda installer could change the display_name in the kernel_spec and install it to a different folder (e.g. xonsh_dev), then I would have three entries in the notebook.

Xonsh
Xonsh (dev)
Xonsh (test)

Any idea how to do this @asmeurer ?

asmeurer · 2015-10-01T15:45:06Z

If you put the absolute path to the install prefix in a text file, conda will automatically replace it with the path of the install prefix (the environment). I don't know if that gets you what you want.

melund · 2015-10-01T18:21:45Z

That is what we use when creating the kernel_spec. That is also the reason that I convert back-slashes to forward slashes, otherwise the prefix is not replaced. What I wanted was to also write the environment name into the file. Right now on my computer, I create the kernel specs manually, so I can access them all from the same notebook.

asmeurer · 2015-10-01T18:41:02Z

I thought the prefix replacement should work with backslashes. I believe you do need to use only one or the other, though (meaning use join(sys.prefix, 'share', 'jupyter', 'kernels', 'xonsh')).

melund · 2015-10-01T20:00:27Z

Hmm. No. The line you are refering to is only used for to create directory in the prefix. The line where backslashes cause problems is sys.prefix in the json file.

spec = {"argv": [sys.executable, 
                         "-m", "xonsh.jupyter_kernel", 
                         "-f", "{connection_file}"],
              "display_name":"Xonsh",
              "language":"xonsh",
              "codemirror_mode":"shell" }

asmeurer · 2015-10-01T20:43:45Z

The backslashes with conda should work. If they aren't you should open an issue in the conda or conda-build issue tracker.

scopatz · 2015-10-03T01:23:32Z

Is there anything that I actually need to do here?

melund · 2015-10-03T07:04:19Z

Sorry. The discussion got derailed. I still can't get the kernel working. Anything else I could try?

scopatz · 2015-10-03T15:51:18Z

What is the problem you are experiencing, @melund? Sorry if it isn't clear to me right now =/

Xonsh

Xonsh (dev)

Xonsh (test)

Any idea how to do this @asmeurer ?

That can be done by changing the label name to be based off the version.

melund · 2015-10-04T07:29:33Z

With pyreadline, everything seems to work fine, but I don't see any output in the notebook. I don't really know what I could easily try.

Running it with prompt toolkit gives me a traceback from the kernel:

  File "C:\Users\mel\Documents\xonsh\xonsh\main.py", line 125, in main_context
    args = premain(argv)
  File "C:\Users\mel\Documents\xonsh\xonsh\main.py", line 72, in premain
    shell = builtins.__xonsh_shell__ = Shell(**shell_kwargs)
  File "C:\Users\mel\Documents\xonsh\xonsh\shell.py", line 40, in __init__
    ctx=self.ctx, **kwargs)
  File "C:\Users\mel\Documents\xonsh\xonsh\prompt_toolkit_shell.py", line 53, in __init__
    enable_search=True, enable_abort_and_exit_bindings=True)
TypeError: __init__() got an unexpected keyword argument 'enable_abort_and_exit_bindings'

Xonsh
Xonsh (dev)
Xonsh (test)
Any idea how to do this @asmeurer ?
That can be done by changing the label name to be based off the version.

I don't think that is the solution with conda packages. Setup.py is not run when installing a conda package. That happens only when the package is build. I think the kernel-spec files should be installed by the conda post-install scripts. We should just wait see. I am sure the conda-team will come up with a good way of doing this.

scopatz · 2015-10-04T16:39:04Z

With pyreadline, everything seems to work fine,

OK. I have pushed up a temporary fix

but I don't see any output in the notebook. I don't really know what I could easily try.

I have noticed that sometimes too with respect to expressions that are evaluated. Maybe we could address this in a future PR?

Also, OK about the conda stuff.

melund · 2015-10-04T18:28:59Z

Sure. If it works on Linux, the rest could be fixed in an other PR. Are there anyone else who can test on windows. I really don't know if it is only my setup...

scopatz · 2015-10-05T19:21:00Z

I can see why this error is occurring, but I don't understand how @melund. I have updated the code. Can you test and see if it is giving you an AssertionError now?

melund · 2015-10-05T19:30:42Z

@scopatz

Traceback (most recent call last):
  File "C:\Users\mel\Documents\xonsh\xonsh\base_shell.py", line 158, in push
    locs=self.ctx)
  File "C:\Users\mel\Documents\xonsh\xonsh\execer.py", line 95, in compile
    tree = self.parse(input, ctx, mode=mode)
  File "C:\Users\mel\Documents\xonsh\xonsh\execer.py", line 70, in parse
    tree = self._parse_ctx_free(input, mode=mode)
  File "C:\Users\mel\Documents\xonsh\xonsh\execer.py", line 151, in _parse_ctx_free
    debug_level=self.debug_level)
  File "C:\Users\mel\Documents\xonsh\xonsh\parser.py", line 264, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "C:\Users\mel\Anaconda3\envs\xonsh\lib\site-packages\ply\yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "C:\Users\mel\Anaconda3\envs\xonsh\lib\site-packages\ply\yacc.py", line 1106, in parseopt_notrack
    p.callable(pslice)
  File "C:\Users\mel\Documents\xonsh\xonsh\parser.py", line 1627, in p_power
    assert False
AssertionError

scopatz · 2015-10-05T19:32:18Z

Is this only in the notebook or on the command line too?

melund · 2015-10-05T19:37:06Z

Only in the notebook. At the command line it works as I would expect.

scopatz · 2015-10-05T19:39:53Z

That is super weird. I know you said that it doesn't matter what you type, but can you give me an example, please? Thanks!

melund · 2015-10-05T19:41:17Z

Here is what I type at the command line to start jupyter:

[xonsh] C:\Users\mel\Documents\xonsh (jupyter)
λ jupyter notebook Untitled5.ipynb
[I 21:38:38.308 NotebookApp] Serving notebooks from local directory: C:\Users\mel\Documents\xonsh
[I 21:38:38.308 NotebookApp] 0 active kernels
[I 21:38:38.309 NotebookApp] The IPython Notebook is running at: http://localhost:8888/
[I 21:38:38.309 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 21:38:40.643 NotebookApp] Kernel started: 1619ae58-37f6-406d-8cc6-6a77d431af56
[IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open'

The UNKNOWN MESSAGE TYPE comes before I provoke the traceback.. So it may not be related.

melund · 2015-10-05T19:49:23Z

Seems that import statements doesn't provoke the error. But everything else I tried does:

scopatz · 2015-10-05T19:52:01Z

Ahh, OK. So I think that this is related. I see the same start up message that you do, but without the [IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open'. What version of jupyter are you using? I have

$ jupyter --version
4.0.6

Also do you have a recent version of ipykernel? The comm_open thing seems to be the correct way that the Jupyter client should act: http://jupyter-client.readthedocs.org/en/latest/messaging.html#opening-a-comm

scopatz · 2015-10-05T19:52:50Z

Also, this is a very weird part of the code base to consistently be failing at...

melund · 2015-10-05T19:58:31Z

I am using jupyter and notebook from continuum's anaconda repo.

[xonsh] C:\Users\mel\Documents\xonsh          
λ jupyter --version                           
4.0.6         
[xonsh] C:\Users\mel\Documents\xonsh (jupyter)
λ jupyter notebook --version
4.0.5

melund · 2015-10-05T20:04:45Z

I have ipykernel version 4.0.3

scopatz · 2015-10-05T20:04:54Z

hmm OK, I was on notebook, version v4.0.4. I updated to v4.0.5 and now I see [IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open'. However, everything seems to run just fine for me.

scopatz · 2015-10-05T20:06:32Z

I am also on ipykernel v4.0.3

scopatz · 2015-10-05T20:12:24Z

Ok I just opened up an issue on ipykernel

scopatz · 2015-10-05T20:22:52Z

Also, what version of ply do you have?

$ python -c "import ply; print(ply.__version__)"

scopatz · 2015-10-05T20:23:14Z

I am on v3.6

melund · 2015-10-05T20:25:35Z

The same as you:
λ python -c "import ply; print(ply.version)"
3.6

scopatz · 2015-10-05T20:28:24Z

Uggh This is nasty. Could you try sticking in a pdb or print statement into bash_shell.BaseShell.default() or .push() to figure out what line it is actually trying to execute? Thanks

melund · 2015-10-05T20:33:03Z

Here I added a print('func:', line ) to both methods.

scopatz · 2015-10-05T20:33:52Z

that makes me sad.

melund · 2015-10-05T20:35:30Z

Well, I am so sorry :)

scopatz · 2015-10-05T20:38:09Z

haha not your fault, of course :)

melund · 2015-10-05T20:44:15Z

It is getting late here, and I am hosting a webinar at work tomorrow. So I will think I will head to bed. I would really like to see if anyone else also have the same issue on windows... Maybe someone who doesn't rely on conda.

scopatz · 2015-10-05T20:44:50Z

good night! I'll post to the mailing list.

scopatz · 2015-11-07T00:15:14Z

I am going to go ahead and merge this with the assumption that some support is better than none. We can build off of this.

scopatz added the feature label Sep 30, 2015

scopatz added this to the v0.3 milestone Sep 30, 2015

asmeurer reviewed Oct 1, 2015
View reviewed changes

scopatz and others added 7 commits October 4, 2015 12:52

partial implementation

6b336c0

basic functionality

13631ea

added completer to notebook

1b1fc1d

Added setting to build for both python 3.4 and 3.5 on anaconda.org

80b0065

Ensure conda build will also install the jupyter kernel spec

7e8b241

Use sys.prefix instead of environment variable.

a504b73

force readline for jupyter

f0cfe74

scopatz force-pushed the jupyter branch from 56341ea to f0cfe74 Compare October 4, 2015 16:52

scopatz added 2 commits October 4, 2015 15:00

docs for jupyter kernel

ca4c565

some fixes for comment only fields

0cb65d7

scopatz mentioned this pull request Oct 5, 2015

[IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open' ipython/ipykernel#64

Open

scopatz mentioned this pull request Oct 16, 2015

ctrl-r not working for reverse search on OS X 10.10.5 with Python 3.5 #394

Closed

fixed minor conflict

25cc8fb

scopatz merged commit 25cc8fb into master Nov 7, 2015

scopatz deleted the jupyter branch November 7, 2015 00:15

Jupyter #399

Jupyter #399

Conversation

scopatz commented Sep 30, 2015

melund commented Sep 30, 2015

melund commented Sep 30, 2015

scopatz commented Sep 30, 2015

melund commented Sep 30, 2015

asmeurer Oct 1, 2015

Choose a reason for hiding this comment

melund Oct 1, 2015

Choose a reason for hiding this comment

asmeurer commented Oct 1, 2015

melund commented Oct 1, 2015

asmeurer commented Oct 1, 2015

melund commented Oct 1, 2015

asmeurer commented Oct 1, 2015

melund commented Oct 1, 2015

asmeurer commented Oct 1, 2015

scopatz commented Oct 3, 2015

melund commented Oct 3, 2015

scopatz commented Oct 3, 2015

melund commented Oct 4, 2015

scopatz commented Oct 4, 2015

melund commented Oct 4, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

scopatz commented Oct 5, 2015

scopatz commented Oct 5, 2015

scopatz commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

melund commented Oct 5, 2015

scopatz commented Oct 5, 2015

scopatz commented Nov 7, 2015