Race condition in weave_rt?

[dabliss]

Hey brian2 team,

I sent an email to the brian2 mailing list about the following warning I'd been getting:

WARNING  brian2.codegen.runtime.weave_rt.weave_rt.failed_compile_test: Cannot use weave, a test compilation failed: invalid syntax (<string>, line 1) (SyntaxError)

It turns out this is pointing to a more serious issue than we originally thought (I think). In addition to these warnings, I'm now getting occasional uncaught SyntaxErrors. All of the errors originate in calls to weave.inline in brian2/codegen/runtime/weave_rt/weave_rt.py.

I've dug into the scipy source code (version 0.15.1), and it appears that calls to inline make use of a set of cache files, which for me have the following paths:

~/.cache/scipy/python27_compiled/linux227compiled_catalog.bak
~/.cache/scipy/python27_compiled/linux227compiled_catalog.dat
~/.cache/scipy/python27_compiled/linux227compiled_catalog.dir

My errors and warnings seem to occur when multiple simultaneous brian2 simulations try to read from and write to these cache files at the same time. Simultaneous writing introduces syntax errors -- i.e., each line is no longer valid python code -- and subsequent reading, followed by an attempted evaluation of the python code, raises the errors.

I'm not sure whether this is a problem for brian2, a problem for scipy, or a problem for me. I suppose what is needed is for use of these cache files to be circumvented. (But wouldn't this come with a speed penalty?)

[thesamovar]

Thanks for looking into this! I've used multiple processes with weave successfully in the past (on Windows) but I think this was probably only using read access to the catalog file. I was generally writing and testing my code on a single process and then doing big runs on multiple processes, so I guess the code was already compiled when I ran the multiple process versions so I never saw this problem. Unless maybe Windows works differently somehow.

[dabliss]

For now I'm switching to cython. This quirk in weave is discussed here and here. One of the discussants came up with the same work-around you used, @thesamovar -- to run a single process all the way through, and then run the others in parallel. That discussion's conclusion, though, was to switch to cython.

[thesamovar]

OK, if you can switch to Cython and the issue doesn't arise there that sounds like a reasonable interim solution. @mstimberg is away at the moment, he might have some ideas when he gets back.

[mstimberg]

This a known limitation of weave, its database is not built for concurrent access. In principle it could work to make weave use different directories for different process (by setting the PYTHONCOMPILED and/or XDG_CACHE_HOME environment variable), but in a quick test I could not get it to work.
I don't think there's much we can do here from the Brian side (except for documenting the issue).

[dabliss]

Many thanks! I'll stick to cython.

[thesamovar]

Weave no longer supported, so closing this.