Using customized lexer.

[BoPeng]

I am developing a kernel for a language that is not available on pygments. It would be nice if Jupyter could accept a Lexer directly from the language definition, such as

 language_info = {
        "pygments_lexer": MyLexer(),
    }

[Carreau]

You need to register your lexer as an entrypoint, then it should work with the alias you have registered it with, as long as your package with the custom lexer is installed.

If this does not work, then it might be a bug.

[BoPeng]

Thank you very much for your suggestion. I use the entrypoint to register my language and I can confirm that the language is registered and can be used from pygments. However, my lexer is never called by jupyter when I select the language (I put some logging message in the constructor of the Lexer).
I actually tried to set my pygments_lexer to python, try to use the echo example from the documentation and see no syntax highlighting either.

I do not know if I have missed anything obvious and I am still checking. If you are interested in trying my code, you can clone

% git clone https://github.com/BoPeng/SOS.git
% cd SOS
% python3 setup.py install

The setup.py process will install both pygments lexer (pysos/sos_show:SoS_Lexer) and jupyter kernel (pysos/kernel.py). Then, after I run jupyter notebook and select language SoS, no syntax highlighting is used (SoS is based on Python 3 so you can type any python expression to test).

[Carreau]

Ah, but the pygments_lexer is use only by nbconvert, the syntax highlight in the notebook is done client side in javascript by codemirror, you need a codemirror mode for live highlight in the notebook.

[BoPeng]

I knew I must have missed something obvious!

Ok, I do not know javascript at all so I have set codemirror_mode to python and got partial syntax highlighting, which is not too bad. Maybe it is time to learn from JS though.

Please let me know if you know some magic (entry point?) to install a customized codemirror mode. I will install a Python.js clone and modify it later.

Thanks,

[Carreau]

Well there are plenty of language supported by codemirror, and the codemirror simplemode which is basically regex/tokentype should be plenty easy to start.

Feel free to ask if you get stuck.

Welcome to the party though, and looking forward to see an announce for your kernel !

[BoPeng]

I spent another few hours on this issue and I think I can implement a codemirror mode for sos. However, before sos becomes official for codemirror/jupyter, what would be the easiest method for users to install the mode so that codemirror and Jupyter can recognize it? I looked through the Jupyter documentation and could not find any information about it. The maxima kernel seems to require users to copy a file to some platform-dependent location which I would like to avoid.

[Carreau]

You can make it an nbextension and enable it. how will depend on which minimal version of notebook you require.

Technically I guess you can drop a kernel.js (it's kind of a hidden feature for kernels developer ) in your kernelspec folder, and it will be loaded with you SOS kernel. Your kernel.js can then just check whether the codemiror SOS mode exists and if not define it.

[BoPeng]

I am afraid that I am still too new to javascript to get it working. I have my spec file installed

$ jupyter kernelspec list
Available kernels:
  python3    /Users/bpeng1/bin/anaconda/lib/python3.5/site-packages/ipykernel/resources
  sos        /usr/local/share/jupyter/kernels/sos

and I put a python clone of .js file there (with python replaced by sos)

$ ls /usr/local/share/jupyter/kernels/sos/
kernel.js   kernel.json

I then fire up jupyter notebook, select SoS kernel, and nothing happens,

$ jupyter notebook
[I 14:39:19.506 NotebookApp] The port 8888 is already in use, trying another random port.
[I 14:39:19.515 NotebookApp] Serving notebooks from local directory: /Users/bpeng1/SOS
[I 14:39:19.515 NotebookApp] 0 active kernels 
[I 14:39:19.515 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/
[I 14:39:19.515 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 14:39:33.596 NotebookApp] Creating new notebook in 
[I 14:39:34.529 NotebookApp] Kernel started: 43a9fff5-5026-403d-b5cd-c371d8f3239c
[IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open'
[W 14:39:35.634 NotebookApp] 404 GET /static/components/codemirror/mode/sos/sos.js?v=20160506143919 (::1) 9.47ms referer=http://localhost:8889/notebooks/Untitled14.ipynb?kernel_name=sos

Because jupyter is looking for /static/components/codemirror/mode/sos/sos.js, I created a folder sos under ~/bin/anaconda/lib/python3.5/site-packages/notebook/static/components/codemirror/mode,
renamed kernel.js to sos.js and put it there. Sure enough, jupyter is happy.

So it looks like jupyter does not load /usr/local/share/jupyter/kernels/sos/kernel.js? I am on a mac using the latest anaconda python3/jupyter. I actually do not mind installing to the system codemirror directory if there is a function similar to install_kernel_spec, or some other way to know where the codemirror directory is.

[BoPeng]

BTW, I also tried to follow the direction in https://carreau.gitbooks.io/jupyter-book/content/kerneljs.html and have

$ cat /usr/local/share/jupyter/kernels/sos/kernel.js
define(function(){

    var onload = function(){
        console.log("I am being loaded")    
    }

    return {onload:onload}
})

and I could not see any I am being loaded message from console when I start Jupyter notebook and start the SoS kernel.

$ jupyter notebook
[I 15:05:10.845 NotebookApp] Serving notebooks from local directory: /usr/local/share/jupyter/kernels/sos
[I 15:05:10.845 NotebookApp] 0 active kernels 
[I 15:05:10.845 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/
[I 15:05:10.845 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 15:05:14.870 NotebookApp] Creating new notebook in 
[I 15:05:15.591 NotebookApp] Kernel started: 582934f9-7769-40ef-94d9-460b7ae726e1
[IPKernelApp] ERROR | UNKNOWN MESSAGE TYPE: 'comm_open'
[W 15:05:16.595 NotebookApp] 404 GET /static/components/codemirror/mode/sos/sos.js?v=20160506150510 (::1) 9.24ms referer=http://localhost:8888/notebooks/Untitled2.ipynb?kernel_name=sos
[W 15:05:43.085 NotebookApp] 404 GET /api/kernels/7ee7f4ec-5883-40b0-aa6c-9d66affa9e20/channels?session_id=6417A317F0D446249C4C144665ADD157 (::1): Kernel does not exist: 7ee7f4ec-5883-40b0-aa6c-9d66affa9e20
[W 15:05:43.087 NotebookApp] 404 GET /api/kernels/7ee7f4ec-5883-40b0-aa6c-9d66affa9e20/channels?session_id=6417A317F0D446249C4C144665ADD157 (::1) 7.11ms referer=None

[Carreau]

Oh, you need to look at the the JS/developer console in your browser to see the messages.

[BoPeng]

Again, many thanks for your quick response. I can see the error message now and should be able to start from there.

[Carreau]

I'm going to assume this has been solved.thanks!

[jma7]

Hi,
I have a similar problem regarding this issue. I did the same thing as mentioned above, put the kernel.js in the kernspec of SOS. When I start a new SOS script, the codemirror works fine, the highlight is correct and there is no error message. However when I open an existing ipynb with SOS script in it, I am getting

http://localhost:8888/static/components/codemirror/mode/sos/sos.js?v=20161212131159 Failed to load resource: the server responded with a status of 404 (Not Found)
notebook.js:115 No CodeMirror mode: sos
notebook.js:116 Require CodeMirror mode error: Error: Script error for "codemirror/mode/sos/sos"

I believe kernel.js is loaded and functioning. So why notebook is looking for sos.js when opening an existing ipynb file? Does this have something to do with the order of js files being loaded?

I am using python 3.5.2 installed in virtualenv, jupyter core 4.2.1, jupyter-client 4.4.0 on MAC OS X.

[BoPeng]

@jma7 please state your os and python environment. I am using anaconda Python 3.5.2, jupyter core 4.2.1, jupyter-client 4.4.0, on Mac OS X and I do not have this problem.

[takluyver]

It might be a race condition - if Codemirror tries to use the sos mode before your custom.js is loaded, then it won't yet be defined. In that case, I guess Codemirror tries to load it as a file from its mode directory.