Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named ipykernel.zmqshell

[garymm]

Steps:

• Clean installation of Ubuntu 18.04
• Install swift-tensorflow v0.11.0 in ~/swift-tensorflow/
• clone this repo, cd into it
• sudo apt-get install python3-venv
• python3 -m venv sju
• . sju/bin/activate
• pip install -r requirements.txt
• python register.py --sys-prefix --swift-toolchain ~/swift-tensorflow/
• mkdir ~/src/tensorflow && cs ~/src/tensorflow
• git clone git@github.com:tensorflow/swift.git
• cd ~/src/tensorflow/swift
• jupyter notebook

In Jupyter web UI:

• navigate to /notebooks/docs/site/tutorials/model_training_walkthrough.ipynb
• Execute all the cells
• This cell fails:

// This cell is here to display the plots in a Jupyter Notebook.
// Do not copy it into another environment.
%include "EnableIPythonDisplay.swift"
IPythonDisplay.shell.enable_matplotlib("inline")

with the error:

Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named ipykernel.zmqshell
Traceback:
  File "/home/garymm/src/swift-jupyter/swift_shell/__init__.py", line 15, in <module>
    from ipykernel.zmqshell import ZMQInteractiveShell
: file PythonKit/Python.swift, line 673
Current stack trace:
0    libswiftCore.so                    0x00007f9e385090a0 swift_reportError + 50
1    libswiftCore.so                    0x00007f9e38574f40 _swift_stdlib_reportFatalErrorInFile + 115
2    libswiftCore.so                    0x00007f9e381defee <unavailable> + 1621998
3    libswiftCore.so                    0x00007f9e381dce00 _assertionFailure(_:_:file:line:flags:) + 827
4    libswiftCore.so                    0x00007f9e38235880 Dictionary.init<A>(_:uniquingKeysWith:) + 0
5    libPythonKit.so                    0x00007f9e390810df <unavailable> + 94431
Current stack trace:
	frame #3: 0x00007f9e38fc44cc $__lldb_expr28`static IPythonDisplay.enable(self=IPythonDisplay) at EnableIPythonDisplay.swift:67:30
	frame #4: 0x00007f9e38fc2057 $__lldb_expr28`main at EnableIPythonDisplay.swift:104:16

Trying to debug a bit, it seems the python being used by the Swift code is not the python from the venv. This code:

let sys = Python.import("sys")
sys.path

Prints:

['/home/garymm/src/swift-jupyter', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

The kernel file looks right to me:

$ cat ./sju/share/jupyter/kernels/swift/kernel.json
{
  "argv": [
    "/home/garymm/src/swift-jupyter/sju/bin/python",
    "/home/garymm/src/swift-jupyter/parent_kernel.py",
    "-f",
    "{connection_file}"
  ],
  "display_name": "Swift",
  "language": "swift",
  "env": {
    "PYTHONPATH": "/home/garymm/swift-tensorflow/usr/lib/python3/dist-packages",
    "LD_LIBRARY_PATH": "/home/garymm/swift-tensorflow/usr/lib/swift/linux",
    "REPL_SWIFT_PATH": "/home/garymm/swift-tensorflow/usr/bin/repl_swift",
    "SWIFT_BUILD_PATH": "/home/garymm/swift-tensorflow/usr/bin/swift-build",
    "SWIFT_PACKAGE_PATH": "/home/garymm/swift-tensorflow/usr/bin/swift-package"
  }
}

[dan-zheng]

Could you try setting the PYTHON_LIBRARY environment variable to point to the libpython.so shared library provided by venv? You can also set PYTHON_LOADER_LOGGING=1 to see debug output for Python library loading.

The "Setting the Python version" section of the Python interoperability tutorial has more details.

[ematejska]

For the notebook, did you use it from the tensorflow-0.11 branch? Since the relesase is not yet final, you have to use the branch.

[garymm]

Thanks, setting PYTHON_LIBRARY before launching seems to have fixed the issue. Perhaps this should be noted in https://github.com/google/swift-jupyter#installation?

[garymm]

I thought about this a bit more, and I think the real confusing thing was that the python interop defaults to using the system python version even when Jupyter is started in a venv. Can the python interop default to using the same version of Python that is running Jupyter? If so, I could re-title this issue with that request.

[garymm]

I'll close the issue and open a new one for what I think would be a nice improvement: #114