Skip to content
This repository

ipython freezes at start if IPYTHONDIR is on an NFS mount #882

Closed
abeld opened this Issue October 16, 2011 · 11 comments

4 participants

abeld Fernando Perez Thomas Kluyver Min RK
abeld

Using ipython 0.11 in virtualenv, I can't start ipython because it freezes, and won't even react to Ctrl-C or 'kill' (has to be killed with 'kill -9')

Running with debug info, I get:

(env)abeld@piko2:0:/l/piko2/d0/abeld/termekhalo_korrelaciok$ ipython --log-level=DEBUG --debug
[TerminalIPythonApp] Config changed:
[TerminalIPythonApp] {'Application': {'log_level': 10}}
[TerminalIPythonApp] Using existing profile dir: u'/loc/remote_computer/d1/abeld/.ipython/profile_default'
[TerminalIPythonApp] Attempting to load config file: ipython_config.py

(/loc/remote_computer/d1 is mounted via NFS but is accessible)

abeld

Most likely the problem is caused by not having pyzmq (or zeromq itself) installed. Since the ipython won't work without that, the real issue is not handling this situation more gracefully.

Thomas Kluyver
Collaborator

Plain terminal IPython should work without ZMQ, and I think we already have an error message when we do need it. I have a feeling it's related to the profile directory being mounted over NFS, but I don't know exactly what the problem is.

Thomas Kluyver
Collaborator

@fperez: Any ideas on this one? I seem to remember other people having a problem when IPYTHONDIR is mounted over the network, but I don't know what the problem is.

abeld

Running ipython under strace, the following is the part where it appears to freeze: (i.e. the last few lines strace prints out)

open("/loc/balin/d1/abeld/.ipython/profile_default/history.sqlite", O_RDWR|O_CREAT, 0644) = 3
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
fstat(3, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
dup(3) = 4
fcntl(3, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=0, len=1}

(/loc/balin/d1/ is mounted via nfs, the history.sqlite file itself is empty.)

Thomas Kluyver
Collaborator

OK, the SQLite FAQ says: "...fcntl() file locking is broken on many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time." We do open two connections to the database (we use a separate thread for writing history, for performance reasons), so perhaps this is the problem.

I don't know what we can do about this. If it raised an error, we could handle it, but if it just blocks, that doesn't work. Is there some reliable way to tell whether the IPYTHONDIR folder is on an NFS mount?

Thomas Kluyver
Collaborator

On #909, @minrk says it works for him with IPYTHONDIR on an NFS mount. I don't know what the key difference is.

Min RK
Owner

Like the SQLite docs say, it affects 'many NFS implementations', not all. And I imagine that it can depend on configuration - my NFS environment was configured quite a bit from the defaults, which may matter.

Fernando Perez
Owner

This is nasty: I'm not quite sure how to detect whether a path is on nfs (I can think of several hacks, but I don't know how reliable/portable/robust they would be). In the long run, I think the solution will have to be to allow the history db to be housed at runtime in a temp directory. IPython would have to copy the db out to a temp directory on startup, use it from the temp directory, and copy it back to its normal place in atexit() operations. Some care will be needed to ensure that with multiple concurrent sessions, only the first copies the database file out so later ones don't clobber it. Handling the fact that sessions can die via segfaults without doing atexit cleanups will be very tricky.

@abeld, in the meantime, your best bet will be to set the IPYTHON_DIR environment variable to point to a non-NFS mount. I realize this is not ideal, as I imagine you're in a shop where $HOME is on NFS. But we hadn't anticipated this clash between NFS and sqlite, and now it's not really easy to back out of our sqlite implementation into a flat file one anymore...

Min RK
Owner

I don't think it's feasible to detect NFS in particular, move the db to a temp location, and restore it after the session. I think we just have to instruct users on NFS to put their history somewhere local (ipython --HistoryManager.hist_file='/tmp/iphist.sqlite' should work, but there is something funky in HistoryAccessor.__init__, preventing config from having any effect. I think it's using its own default mechanism, so it effectively ignores configuration. I would imagine that a symlink to outside the NFS mount should also work.

What would be helpful is if we can more directly detect a hang in the right place, and then show a message, explaining how to put their history off of the NFS mount.

@takluyver - Am I right in assuming that it is the init_db call that would be hanging? Perhaps we can detect a hang there with SIGALRM (though it may be a locked up call that won't respond to the signal).

Thomas Kluyver
Collaborator

I think it's the sqlite3.connect call hanging, yes. The trace above shows an open() call at the C level.

Fernando Perez
Owner

Guys, in light of #909 and #940 having been merged, I'm closing this, as I don't think there's anything more we can do. The nfs limitations of sqlite are beyond our power to fix, and now we have alternatives thanks to these two PRs, so there really isn't anything else we can do here. Unfortunately, users whose IPYTHON_DIR is on an NFS mount that exhibits this problem, will need to manually config their hist files to reside elsewhere.

Fernando Perez fperez closed this November 29, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.