Running scipy.weave programs on cluster with shared filesystem (Trac #1370)

[scipy-gitbot]

Original ticket http://projects.scipy.org/scipy/ticket/1370 on 2011-01-23 by trac user temporaer, assigned to unknown.

my application makes use of scipy.weave (0.4.9) to run c-code on numpy arrays. The same function is called with different data on multiple computers of our cluster using the parallel execution mechanisms coming with ipython.

As far as I can see, weave stores its compiled libraries in ~/.python26_compiled. As all computers share the file system via NFS, this will actually be the same folder, and the same library file name, resulting in write conflicts (and /weird/ error messages)

I put a symbolic link from /tmp to ~/.python26_compiled and that seemed to fix the problem.

Should there be a way to specify and create directories elsewhere and/or include the hostname in the hash-generation? Or am I missing something important?

[dabliss]

i'm confused about why putting a symbolic link from /tmp to ~/.python26_compiled would fix the problem. can you explain that?

[mstimberg]

@dabliss: I think the OP meant to say that they linked ~/.python26_compiled to /tmp -- probably the latter is on a local drive for each machine and not on the shared NFS.

[person142]

Closing since weave is no longer a part of SciPy. This issue is now tracked at scipy/weave#7.