Kernel crashes when importing tensorflow #7600

Napuu · 2021-09-21T17:03:55Z

Environment data

VS Code version: 1.60.1
Jupyter Extension version (available under the Extensions sidebar): v2021.8.2041215044
Python Extension version (available under the Extensions sidebar): v2021.9.1246542782
OS (Windows | Mac | Linux distro) and version: debian 11
Python and/or Anaconda version: 3.9.2
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): none
Jupyter server running: Local (I think, running notebook on a vm with ssh extension)

Expected behaviour

import tensorflow would work

Actual behaviour

See screenshot below

Steps to reproduce:

Initially I got same error messages as here #7190, but unfortunately can't reproduce them anymore.

Error: Kernel is dead
Error: Kernel is dead
at g._sendKernelShellControl (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:1006195)
at g.sendShellMessage (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:1005964)
at g.requestExecute (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:1008506)
at d.requestExecute (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:37:325680)
at w.requestExecute (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:32:18027)
at w.executeCodeCell (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:301076)
at w.execute (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:300703)
at w.start (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:296367)
at async t.CellExecutionQueue.executeQueuedCells (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:311160)
at async t.CellExecutionQueue.start (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:310700)

The text was updated successfully, but these errors were encountered:

Napuu · 2021-09-21T19:14:51Z

Eh, apparently this has nothing to do with jupyter. Following message is returned when trying to import tensorflow on the same machine outside jupyter:

The TensorFlow library was compiled to use SSE4.1 instructions, but these aren't available on your machine.

Error message could be better at jupyter side, but feel free to close this issue if you think otherwise.

IanMatthewHuff · 2021-09-21T22:50:04Z

@Napuu Thanks for the follow up. Given the message that you were seeing this is something that we won't be able to handle. I'm going to leave this issue open for discussion in our triage as I agree that the error message here is not good. Might be something we can do better here. Out of interest, if you look at the "Jupyter" tab of the output window do you see the actual SSE4.1 error message anywhere in that output?

Napuu · 2021-09-22T14:59:13Z

Unfortunately no. Just generic errors about kernel dying.

DonJayamanne · 2021-09-23T22:44:30Z

@Napuu Please could you provide the full log from teh Jupyter output panel.
I'd like to see if we detected the failure earlier on (this way we can notify the user in a better way)

Napuu · 2021-09-28T09:25:23Z

@DonJayamanne sure. Here is log after restarting and running cell with content import tensorflow as tf

Info 2021-09-28 12:22:44: Execute Cells request 1 1
Info 2021-09-28 12:22:44: Execute Cell 1 file:///home/debian/building-classifier/tensorflow-test.ipynb
Info 2021-09-28 12:22:45: KernelProcess output: Content-Length: 128
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "method": "log", "params": {"level": "WARNING", "msg": "Kernel has exited with exit code -4, ", "pid": 3968}}
Warn 2021-09-28 12:22:45: Python Daemon (pid: 3968): Kernel has exited with exit code -4, 
Info 2021-09-28 12:22:45: KernelProcess output: Content-Length: 86
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "method": "kernel_died", "params": {"exit_code": -4, "reason": ""}}
Error 2021-09-28 12:22:45: Kernel died [s [Error]: Kernel died with exit code -4. /home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.
  warn(

	at /home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:787065
	at /home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:462397
	at /home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:462615
	at Immediate._onImmediate (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:464180)
	at processImmediate (internal/timers.js:461:21)] {
  category: 'kerneldied',
  stdErr: `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.\n` +
    '  warn(\n' +
    `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.\n` +
    '  warn(\n',
  exitCode: -4,
  reason: `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.\n` +
    '  warn(\n' +
    `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.\n` +
    '  warn(\n'
} /home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.
  warn(

Error 2021-09-28 12:22:45: KernelProcess Exit Exit - -4, /home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.
  warn(
 [s [Error]: Kernel died with exit code -4. /home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.
  warn(

	at /home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:787065
	at /home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:462397
	at /home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:462615
	at Immediate._onImmediate (/home/debian/.vscode-server/extensions/ms-toolsai.jupyter-2021.8.2041215044/out/client/extension.js:52:464180)
	at processImmediate (internal/timers.js:461:21)] {
  category: 'kerneldied',
  stdErr: `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.\n` +
    '  warn(\n' +
    `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.\n` +
    '  warn(\n',
  exitCode: -4,
  reason: `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.\n` +
    '  warn(\n' +
    `/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.\n` +
    '  warn(\n'
}
Error 2021-09-28 12:22:45: Disposing session as kernel process died ExitCode: -4, Reason: /home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.
  warn(

Info 2021-09-28 12:22:45: Dispose Kernel process
Info 2021-09-28 12:22:45: kill daemon
Info 2021-09-28 12:22:45: KernelProcess output: Content-Length: 131
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "method": "log", "params": {"level": "INFO", "msg": "Execute rpc method kill_kernel in DS Daemon", "pid": 3968}}Content-Length: 167
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "method": "log", "params": {"level": "INFO", "msg": "Execute rpc method kill_kernel from /home/debian/miniconda3/envs/tf9/bin/python", "pid": 3968}}Content-Length: 128
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "method": "log", "params": {"level": "INFO", "msg": "Kill kernel in DS Kernel Launcher Daemon", "pid": 3968}}Content-Length: 138
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "method": "log", "params": {"level": "INFO", "msg": "Kill kernel in DS Kernel Launcher Daemon, pid 3969", "pid": 3968}}Content-Length: 43
Content-Type: application/vscode-jsonrpc; charset=utf8

{"jsonrpc": "2.0", "id": 2, "result": null}
Info 2021-09-28 12:22:45: Python Daemon (pid: 3968): Execute rpc method kill_kernel in DS Daemon
Info 2021-09-28 12:22:45: Python Daemon (pid: 3968): Execute rpc method kill_kernel from /home/debian/miniconda3/envs/tf9/bin/python
Info 2021-09-28 12:22:45: Python Daemon (pid: 3968): Kill kernel in DS Kernel Launcher Daemon
Info 2021-09-28 12:22:45: Python Daemon (pid: 3968): Kill kernel in DS Kernel Launcher Daemon, pid 3969
Info 2021-09-28 12:22:45: Dispose Kernel process
Info 2021-09-28 12:22:45: KernelProcess Exit Exit - -4 /home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
/home/debian/.local/lib/python3.9/site-packages/traitlets/traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use 'e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb' instead of 'b"e8ea1d9e-6818-4f4e-9382-2f4017fe3ddb"'.
  warn(

DonJayamanne · 2021-09-28T15:49:49Z

@Napuu thanks

Please could you provide the following:
versions of all packages install, python -m pip list
- I'd like to replicate this at my end so we can handle such cases and provide more meaningful messages

FYI - my suggestion is to create another environment or re-isntall tensorflow to remedy this issue.

Napuu · 2021-09-28T16:03:59Z

@DonJayamanne here you go
I created this env with conda create -n tf tensorflow

python -m pip list
Package                      Version
---------------------------- -------------------
absl-py                      0.13.0
affine                       2.3.0
aiohttp                      3.7.4.post0
astor                        0.8.1
astunparse                   1.6.3
async-timeout                3.0.1
attrs                        21.2.0
backcall                     0.2.0
blinker                      1.4
brotlipy                     0.7.0
cached-property              1.5.2
cachetools                   4.2.2
certifi                      2021.5.30
cffi                         1.14.6
chardet                      4.0.0
charset-normalizer           2.0.4
clang                        5.0
click                        8.0.1
click-plugins                1.1.1
cligj                        0.7.2
cloudpickle                  2.0.0
coverage                     5.5
cryptography                 3.4.7
cycler                       0.10.0
Cython                       0.29.24
debugpy                      1.4.3
decorator                    5.1.0
dm-tree                      0.1.6
entrypoints                  0.3
flatbuffers                  1.12
gast                         0.4.0
google-auth                  1.35.0
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
grpcio                       1.40.0
h5py                         3.1.0
idna                         3.2
importlib-metadata           4.8.1
ipykernel                    6.4.1
ipython                      7.27.0
ipython-genutils             0.2.0
jedi                         0.18.0
jupyter-client               7.0.3
jupyter-core                 4.8.1
keras                        2.6.0
keras-nightly                2.7.0.dev2021092107
Keras-Preprocessing          1.1.2
kiwisolver                   1.3.2
libclang                     11.1.0
Markdown                     3.3.4
matplotlib                   3.4.3
matplotlib-inline            0.1.3
mkl-fft                      1.3.0
mkl-random                   1.2.2
mkl-service                  2.4.0
multidict                    5.1.0
nest-asyncio                 1.5.1
numpy                        1.19.5
oauthlib                     3.1.1
opencv-python                4.5.3.56
opt-einsum                   3.3.0
parso                        0.8.2
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       8.3.2
pip                          21.2.4
prompt-toolkit               3.0.20
protobuf                     3.18.0
psycopg2-binary              2.9.1
ptyprocess                   0.7.0
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycosat                      0.6.3
pycparser                    2.20
Pygments                     2.10.0
PyJWT                        2.1.0
pyOpenSSL                    20.0.1
pyparsing                    2.4.7
PySocks                      1.7.1
python-dateutil              2.8.2
pyzmq                        22.3.0
rasterio                     1.2.8
requests                     2.26.0
requests-oauthlib            1.3.0
rsa                          4.7.2
ruamel.yaml                  0.17.16
ruamel.yaml.clib             0.2.6
scipy                        1.7.1
setuptools                   58.0.4
six                          1.15.0
snuggs                       1.4.7
tb-nightly                   2.7.0a20210921
tensorboard                  2.6.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.0
tensorflow                   2.6.0
tensorflow-estimator         2.6.0
tensorflow-hub               0.12.0
tensorflow-io-gcs-filesystem 0.21.0
tensorflow-probability       0.14.0
termcolor                    1.1.0
tf-estimator-nightly         2.7.0.dev2021092108
tornado                      6.1
traitlets                    5.1.0
typing-extensions            3.7.4.3
urllib3                      1.26.6
wcwidth                      0.2.5
Werkzeug                     2.0.1
wheel                        0.37.0
wrapt                        1.12.1
yarl                         1.6.3
zipp                         3.5.0

On my machine I got it working by switching cpu from kvm64 -> host, as this instance is in VM.

DonJayamanne · 2021-10-12T18:19:37Z

Note:

Issue has been resolved
TODO: install same environment locally and see if we can replicate this and provide better error messagse to the user

DonJayamanne · 2021-10-12T18:25:03Z

@Napuu thanks for the information, i've tried to repro this and unfortunately i cannot.
CLosing as you've managed to get this working.

Napuu added the bug Issue identified by VS Code Team member as probable bug label Sep 21, 2021

github-actions bot added the needs-triage label Sep 21, 2021

github-actions bot assigned IanMatthewHuff Sep 21, 2021

IanMatthewHuff removed their assignment Sep 21, 2021

IanMatthewHuff added notebook-workflow Issues that interrupt expected or desirable behavior papercut 🩸 Something affecting the productivity of the team labels Sep 21, 2021

greazer added kernel-crash and removed needs-triage labels Sep 23, 2021

greazer added this to the October 2021 milestone Sep 23, 2021

DonJayamanne self-assigned this Oct 12, 2021

DonJayamanne closed this as completed Oct 12, 2021

DonJayamanne mentioned this issue Oct 12, 2021

Better handling of kernel crashes (dead kernels) #7846

Closed

greazer added the verified Verification succeeded label Oct 27, 2021

DonJayamanne mentioned this issue Feb 21, 2022

Summary of Kernel Issues #7901

Closed

1 task

DonJayamanne mentioned this issue Mar 15, 2022

Can we deal with user code (packages) causing kernel failures #9375

Closed

github-actions bot locked as resolved and limited conversation to collaborators Oct 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kernel crashes when importing tensorflow #7600

Kernel crashes when importing tensorflow #7600

Napuu commented Sep 21, 2021

Napuu commented Sep 21, 2021

IanMatthewHuff commented Sep 21, 2021

Napuu commented Sep 22, 2021

DonJayamanne commented Sep 23, 2021

Napuu commented Sep 28, 2021

DonJayamanne commented Sep 28, 2021

Napuu commented Sep 28, 2021

DonJayamanne commented Oct 12, 2021

DonJayamanne commented Oct 12, 2021

Kernel crashes when importing tensorflow #7600

Kernel crashes when importing tensorflow #7600

Comments

Napuu commented Sep 21, 2021

Environment data

Expected behaviour

Actual behaviour

Steps to reproduce:

Napuu commented Sep 21, 2021

IanMatthewHuff commented Sep 21, 2021

Napuu commented Sep 22, 2021

DonJayamanne commented Sep 23, 2021

Napuu commented Sep 28, 2021

DonJayamanne commented Sep 28, 2021

Napuu commented Sep 28, 2021

DonJayamanne commented Oct 12, 2021

Note:

DonJayamanne commented Oct 12, 2021