Jupyter won't open or trust notebooks that worked previously

[moble]

I'm having a problem with both jupyter notebook and just plain jupyter trust, which both seem to have the same underlying issue. Notebooks that were working previously are no longer working. I had updated everything with conda recently, but I tried just totally reinstalling conda (python 3.5) and get the same result.

I've found this ipython issue, which is slightly different (different error message), but in any case the solution to that doesn't solve this. In fact, I tried completely removing my ipython and jupyter profile directories, and nothing changed. I also found this traitlets issue that at least reports the same exception, but that was fixed a few months ago, so it shouldn't be an issue, right??? Finally, there's also this issue, but all my directories have the correct permissions and ownership.

If I try to load the notebooks, I see a pop-up saying

Error loading notebook

An unknown error occurred while loading this notebook. This version can load notebook formats v4 or earlier. See the server log for details.

and the log says

[E 13:44:38.278 NotebookApp] Unhandled error in API request
    Traceback (most recent call last):
      File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/traitlets.py", line 501, in get
        value = obj._trait_values[self.name]
    KeyError: 'db'

    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/home/myuser/anaconda/lib/python3.5/site-packages/notebook/base/handlers.py", line 457, in wrapper
        result = yield gen.maybe_future(method(self, *args, **kwargs))
      File "/home/myuser/anaconda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
        value = future.result()
      File "/home/myuser/anaconda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
      File "/home/myuser/anaconda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
        yielded = next(result)
      File "/home/myuser/anaconda/lib/python3.5/site-packages/notebook/services/contents/handlers.py", line 124, in get
        path=path, type=type, format=format, content=content,
      File "/home/myuser/anaconda/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 358, in get
        model = self._notebook_model(path, content=content)
      File "/home/myuser/anaconda/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 318, in _notebook_model
        self.mark_trusted_cells(nb, path)
      File "/home/myuser/anaconda/lib/python3.5/site-packages/notebook/services/contents/manager.py", line 447, in mark_trusted_cells
        trusted = self.notary.check_signature(nb)
      File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 220, in check_signature
        if self.db is None:
      File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/traitlets.py", line 529, in __get__
        return self.get(obj, cls)
      File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/traitlets.py", line 508, in get
        value = self._validate(obj, dynamic_default())
      File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 127, in _db_default
        self.init_db(db)
      File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 139, in init_db
        )""")
    sqlite3.OperationalError: disk I/O error
[E 13:44:38.289 NotebookApp] {
      "X-Requested-With": "XMLHttpRequest",
      "Referer": "https://localhost:6003/notebooks/MyNotebook.ipynb",
      "Dnt": "1",
      "Connection": "keep-alive",
      "Accept-Encoding": "gzip, deflate, sdch, br",
      "Cookie": "username-localhost-6003=2|1:0|10:1466185422|23:username-localhost-6003|48:NDI5NTg3ZjItMzE4ZS00N2ZjLWEyNjItNjI2OGZlYWNjMWVh|c528ffaedf4e4f72cd802cc1abd197a38d1743cedacd923483208441a870664d",
      "Accept-Language": "en-US,en;q=0.8",
      "Host": "localhost:6003",
      "Accept": "application/json, text/javascript, */*; q=0.01",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36"
    }
[E 13:44:38.289 NotebookApp] 500 GET /api/contents/MyNotebook.ipynb?type=notebook&_=1466185495467 (::1) 346.33ms referer=https://localhost:6003/notebooks/MyNotebook.ipynb

If I try to trust the notebooks from the command line, I see something like this:

> jupyter trust MyNotebook.ipynb
Traceback (most recent call last):
  File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/traitlets.py", line 501, in get
    value = obj._trait_values[self.name]
KeyError: 'db'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/myuser/anaconda/bin/jupyter-trust", line 8, in <module>
    main()
  File "/home/myuser/anaconda/bin/jupyter-trust", line 5, in main
    TrustNotebookApp.launch_instance()
  File "/home/myuser/anaconda/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/config/application.py", line 596, in launch_instance
    app.start()
  File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 416, in start
    self.sign_notebook(notebook_path)
  File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 393, in sign_notebook
    if self.notary.check_signature(nb):
  File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 220, in check_signature
    if self.db is None:
  File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/traitlets.py", line 529, in __get__
    return self.get(obj, cls)
  File "/home/myuser/anaconda/lib/python3.5/site-packages/traitlets/traitlets.py", line 508, in get
    value = self._validate(obj, dynamic_default())
  File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 127, in _db_default
    self.init_db(db)
  File "/home/myuser/anaconda/lib/python3.5/site-packages/nbformat/sign.py", line 139, in init_db
    )""")
sqlite3.OperationalError: disk I/O error

Any workaround or fix for this?

[moble]

I also created a python 2.7 conda env, and I get the same error.

[takluyver]

The file it's trying to open is $(jupyter --data-dir)/nbsignatures.db - try moving that aside and then try again.

Is it possible that your home directory is on an NFS mount? That seems to be the most common cause of issues with sqlite databases. We try to catch these errors and automatically recreate the database file, but maybe we're missing this one.

[moble]

The file it's trying to open is $(jupyter --data-dir)/nbsignatures.db - try moving that aside and then try again.

Nope, that's one of the ones I tried moving. Tried again just now to make sure, and it didn't help. The file did get created again, but echo '.dump' | sqlite3 $(jupyter --data-dir)/nbsignatures.db gives me

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
/**** ERROR: (10) disk I/O error *****/
ROLLBACK; -- due to errors

Is it possible that your home directory is on an NFS mount? That seems to be the most common cause of issues with sqlite databases. We try to catch these errors and automatically recreate the database file, but maybe we're missing this one.

It appears to be Lustre [that's what I get when I do stat -f -L -c %T $(jupyter --data-dir)], which I can believe would have problems similar to those on NFS.

Anything I can do to help diagnose this? Or maybe just workaround signatures?

[takluyver]

Try configuring the notebook to store the trust database on a local scratch disk, or if that's not possible, the special :memory: location for sqlite.

The config option is NotebookNotary.db_file, and if you're not familiar with the config file, here are the docs: http://jupyter-notebook.readthedocs.io/en/latest/config.html

[moble]

Thanks, that works.

Would it help to print a warning if a sqlite3.OperationalError occurs during self.init_db inside nbformat/sign.py? Maybe suggesting this solution?

[takluyver]

I think it could do something like we do with the IPython history database: if it fails once, move the database aside and try to recreate it (it already does this). If recreating it fails, switch to an in-memory database and give a warning.

[moble]

if it fails once, move the database aside and try to recreate it (it already does this)

Ah, I see that's been in master for the past 5 months, but not in the version I currently get from conda.

If recreating it fails, switch to an in-memory database and give a warning.

I'm just creating a PR to try this.

[moble]

This error should be handled by the newest version of nbformat.

[minrk]

Thanks!

[abhimanyu3-zz]

I was getting the error because my disk was full. I moved some of the stuff and its fine now.

[iderevitskiy]

For me, removing nbsignatures.db from directory '~/.local/share/jupyter' is work.

[marctorsoc]

This happened to me recently with old notebooks, and also wasn't able to create new notebooks. There were two problems:

a) couldn't open old notebooks (unhandled error)
b) couldn't save notebooks
c) couldn't create new notebooks (sqlite problem, database locked)

I did two things:

a) vim .jupyter/jupyter_notebook_config.py and then c.NotebookNotary.db_file = ':memory:'
b) opening some of the notebooks with pycharm and save. Maybe just with (a) it works, but before I found (a) this allowed me to open notebooks in jupyter (but not to save them)

Now everything seems to work :)

[prhbrt]

This still doesn't work on NFS mounts for:

jupyter core     : 4.6.1
jupyter-notebook : 6.0.3
qtconsole        : not installed
ipython          : 7.11.1
ipykernel        : 5.1.4
jupyter client   : 5.3.4
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : not installed
nbformat         : 5.0.4
traitlets        : 4.3.3

[nathancarter]

I have encountered this error intermittently when running Jupyter in an official Docker image, specifically, with this command:

docker run -p 8888:8888 \
           -v "$THE_FOLDER_I_CARE_ABOUT:/home/jovyan" \
           -e JUPYTER_ENABLE_LAB=yes \
           --name "jupyter-lab-$SIMPLE_VERSION_OF_THE_FOLDER_NAME" \
           --rm \
           jupyter/datascience-notebook

The error appears on the console when attempting to open an .ipynb file, but what happens in the browser UI for Jupyter is that the moment you open the file, it immediately asks if you're sure you want to close it without saving (even before it has shown you the file). If you say yes, it closes it without ever successfully opening it. If you say no, it just spins a progress spinner in the center of the relevant tab indefinitely.

As of this point, it's not clear to me exactly what causes me to sometimes have the error occur and sometimes not have it occur. Sometimes opening a file causes the problem and sometimes it doesn't. Restarting the container seems to help.

I'm on OS X using Chrome and Docker (as opposed to podman, etc.), in case those details are relevant. The particular image version (if I'm doing this right, with docker image history) seems to be ec2a10ca40d6, which is from about 2 months ago.

[nathancarter]

Just confirmed: When I get this error, shutting down the container and restarting it lets me open the file without errors.