Unable to open a second instance of IPython

[jni]

I am getting the following error when trying to open a new iPython instance:

 $ ipython --pylab
Traceback (most recent call last):
  File "/usr/bin/ipython", line 9, in <module>
    load_entry_point('ipython==0.12', 'console_scripts', 'ipython')()
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/frontend/terminal/ipapp.py", line 402, in launch_new_instance
    app.initialize()
  File "<string>", line 2, in initialize
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/config/application.py", line 84, in catch_config_error
    return method(app, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/frontend/terminal/ipapp.py", line 312, in initialize
    self.init_shell()
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/frontend/terminal/ipapp.py", line 332, in init_shell
    ipython_dir=self.ipython_dir)
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/config/configurable.py", line 318, in instance
    inst = cls(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/frontend/terminal/interactiveshell.py", line 183, in __init__
    user_module=user_module, custom_exceptions=custom_exceptions
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/core/interactiveshell.py", line 431, in __init__
    self.init_history()
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/core/interactiveshell.py", line 1439, in init_history
    self.history_manager = HistoryManager(shell=self, config=self.config)
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/core/history.py", line 407, in __init__
    self.new_session()
  File "<string>", line 2, in new_session
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/core/history.py", line 61, in needs_sqlite
    return f(*a,**kw)
  File "/usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/IPython/core/history.py", line 425, in new_session
    NULL, "") """, (datetime.datetime.now(),))
OperationalError: database is locked

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@scipy.org

This happens when the other iPython session is executing a particularly lengthy command, but not when it is idle. When I try to run something inside an already open session while a command is running in a separate session, I get:

The history saving thread hit an unexpected error (OperationalError('database is locked',)).History will not be written to the database.

The cause is obvious, but it seems to me that the functionality is broken... History commands generated while the db is locked should probably be added to a queue to be submitted once the lock is removed.

[minrk]

I can't seem to trigger this on purpose, but I think I understand what could cause the issue. What platform are you on, and is your home directory mounted via NFS or other network share?

[fperez]

I suspect it's a lock on the sqlite history database, but I've also never seen it. @takluyver, have you ever encountered this one?

[jni]

@minrk is right: my home is on NFS. I'm running Fedora Core 16.

[fperez]

This is your problem then: http://www.sqlite.org/faq.html#q5. SQLite locking doesn't work reliably on NFS filesystems.

Your best bet is to start ipython directing it to use a local file for history:

ipython --HistoryManager.hist_file=/tmp/ipython_hist.sqlite

Just use a path that is not NSF mounted. You can set this variable permanently in your config file if you find a value that works well.

I don't know that we can really do anything else on our end, since the problem is really beyond ipython's reach.

[jni]

Ah. Sounds good. In the config file, is it:

c = get_config()
c.HistoryManager.hist_file='/tmp/ipython_hist.sqlite'

?

Thanks!

[takluyver]

On 29 January 2012 23:41, Juan Nunez-Iglesias <
reply@reply.github.com

wrote:

Ah. Sounds good. In the config file, is it:

c = get_config()
c.HistoryManager.hist_file='/tmp/ipython_hist.sqlite'

That should be it, yes.

Thomas

[ludwigschwardt]

I also ran into this issue...

I instantiate an InteractiveShell object during module initialisation in order to register a custom tab completer:

from IPython.core.interactiveshell import InteractiveShell
ip = InteractiveShell.instance()
ip.set_hook(...)

This module is imported by multiple processes that start up in parallel, leading to a race condition where some of the processes may hang on the reported OperationalError exception. I am using an SQLite history database on an HFS+ filesystem on a Mac (so no NFS problems here).

Any ideas or workarounds would be welcomed.

[minrk]

If you don't actually need to access the history, you can override the config to use an in-memory database:

from IPython.config.loader import Config
from IPython.core.interactiveshell import InteractiveShell

cfg = Config()
cfg.HistoryManager.hist_file = ':memory:'
ip = InteractiveShell.instance(config=cfg)

which presumably wouldn't have multiprocessing issues.

[ludwigschwardt]

Thanks,

That's very useful to know. In the end I opted to just catch the OperationalError exception and ignore the tab completer in that case, as this race condition typically occurs in system processes with no need for user interaction.

The problem is that our code supports IPython 0.10, 0.11 and 0.12, and your code seems to be 0.12 only (if I consider your pull request 940). I already have two code paths for 0.10 and 0.11+ and want to avoid yet another code path...

[takluyver]

Well, using :memory: will presumably create a separate in-memory database in each process that instantiates InteractiveShell, so it should avoid any contest.

How many processes do you have starting up in parallel? I'm a bit surprised it has a problem with this, but maybe if there are a lot of processes, it's just the sheer number trying to access the database that causes it.

[ludwigschwardt]

We have in the order of 10 processes starting up. Every now and then (more then than now, it seems), 2-3 processes will lock up with this error. Of course, I cannot reproduce it now after several startup attempts in order to verify this :-)