ipython crashes with the following message. I do not what went wrong. Can you help me identify the problem?

[ajithpad]

Traceback (most recent call last):
File "/usr/bin/ipython", line 8, in
launch_new_instance()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 402, in launch_new_instance
app.initialize()
File "", line 2, in initialize
File "/usr/lib/python2.7/dist-packages/IPython/config/application.py", line 84, in catch_config_error
return method(app, _args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 312, in initialize
self.init_shell()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 332, in init_shell
ipython_dir=self.ipython_dir)
File "/usr/lib/python2.7/dist-packages/IPython/config/configurable.py", line 318, in instance
inst = cls(_args, *_kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/interactiveshell.py", line 183, in init
user_module=user_module, custom_exceptions=custom_exceptions
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 431, in init
self.init_history()
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 1439, in init_history
self.history_manager = HistoryManager(shell=self, config=self.config)
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 409, in init
self.new_session()
File "", line 2, in new_session
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 63, in needs_sqlite
return f(a,*kw)
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 428, in new_session
self.session_number = cur.lastrowid
OperationalError: database is locked

[takluyver]

Did you have another copy of IPython available at the time? Is your home directory on a network folder?

[ajithpad]

No I did not have any other copy of ipython running at the time. Yes, my home directory is part of the network folder.

[takluyver]

OK, that will be the problem - IPython stores history in an sqlite database, and sqlite has problems with locking files on the network. We haven't found any good way to automatically detect whether a folder is on a network drive or not.

To work around it: look for the file ipython_config.py in the location given by ipython locate profile. If it doesn't exist, run ipython profile create to make it. Within that file, search for c.HistoryManager.hist_file, and set it to something on the local filesystem (it can be in '/tmp' if you're not interested in your command history. If nothing on the local system is writeable, set it to ':memory:' to use an in-memory database.

[ajithpad]

Thank you. But two issues

1. In the ipython_config.py (which existed) the search for c.HistoryManager.hist_file yields no results.
2. Apart from c = get_config(), everything else is commented out in the file. Is something wrong there?

[takluyver]

Hmm, what version of IPython are you using? You could try ipython profile create --reset to make new default config files.

Yes, everything in the file is commented out by default, so that we can
change the defaults in later releases. Uncomment anything you want to
change.

[ajithpad]

I use version 2.7.3 . ipython profile create --reset also did not also create the file with the c.HistoryManager.hist_file. Is there some work around? Thanks for your help :)

[takluyver]

That's your Python version. IPython's version number will probably be 0.12,
0.12.1 or 0.13.

If you search through the file for 'hist', do you see any similar entries?

Another possible workaround is to set the IPYTHONDIR environment variable
to a folder on the local filesystem.

[ajithpad]

I used export IPYTHONDIR = /tmp and it worked. I made changes to reflect the same in my .bashrc. But I still don't understand why it worked? Thank you so much for the help and time :)

[takluyver]

IPython puts all your files, including the history db, into a directory -
normally .config/ipython or .ipython, but if you specify IPYTHONDIR, it
uses that instead. If it's on a network folder, the history database has
the problem you already saw.

The downside of setting IPYTHONDIR to /tmp is that you can't permanently
set any config, because it looks for config in that folder.

I'll close this issue, because we already know about the problem, but we
don't think there's any good way to fix it.

[ajithpad]

Thank you once again..

[bfroehle]

We store the history of commands you type into IPython in a SQLite database in the folder @takluyver mentioned earlier. To prevent two processes from accidentally writing to the SQLite database file at the same time, the file is locked --- meaning only one process can edit the file at a time.

Normally this file locking is handled by the operating system in a robust way. However there are some known issues with SQLite on NFS drives. This might be what you are experiencing.

You can read more about this error: SQLite: Database Is Locked.

The export IPYTHONDIR=/tmp works by placing a new history database (history.sqlite) in the /tmp directory which is presumably local and for which file locking works properly. You might also try to delete the SQLite history database in your home directory, or move it away and copy it back (as the previous link mentions), in an attempt to manually "unlock" the stuck file.

[bfroehle]

@takluyver Do we have a page on our wiki for this bug? It does seem to come up from time to time. It might be worth catching this error and referencing a wiki page to help users get un-stuck from this bug.

[takluyver]

Good idea, that sounds worthwhile. The only issue is that I don't know if the error appears in a consistent place - but we can just catch it in this location, and add catches when other tracebacks are reported. If you've got time to haev a go at this, you're welcome to.

[amelio-vazquez-reina]

Hi all @bfroehle I am running into this problem too. I sometimes have to spawn hundreds of jobs on a cluster all running the same code (on different datasets). Part of this code does the following:

ipshell = InteractiveShellEmbed(config=cfg, banner1=banner_msg, exit_msg=exit_msg)

(but it does not start the shell)

WIth so many jobs running in parallel many of them attempt to access a SQLite that is locked at the same time (this is all on a network drive), which seems to cause the Python script running the code above to exit abruptly.

Is there a way to have IPython wait until the SQlite database is unlocked rather than abruptly exiting if is locked? I guess that as you all said, this is handled by the OS and there is no way to circumvent this, correct?

As for the export IPYTHONDIR = /tmp trick, what would happen if multiple Python processes attempt to access SQlite simultaneously on the same machine? Would the OS handle this situation properly?

[minrk]

As for the export IPYTHONDIR = /tmp trick, what would happen if multiple Python processes attempt to access this file simultaneously from the same machine?

It should be fine on a normal filesystem. The problem is generally that the way the locking / coordination is implemented is often broken when accessed over NFS.