Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Add multi-process support #66
referenced this pull request
Sep 27, 2016
Oct 10, 2016
FWIW, I started to work on performance improvement of your branch last week and simply went with JSON instead of shelve/BSDdb, which also gave an improvement of a few orders of magnitude. (I don't know the exact number right now but it was somewhere around a 2x slowdown compared to non-multiprocessing.) Maybe that's an alternative that involves less custom code.
That's about the same slowdown as this code, which is presumably dominated by the 2 mutexes in use (why does using a mutex take most of a microsecond?). Hopefully we'll get that faster some time, but I doubt you got too much better than the 80 lines of code I managed it in.
Sorry, I did not mean to say that your code was bad or something. Actually, the binary format is very simple and the code is easy to understand. My point is that if it does not make a difference in terms of performance, maybe it's more future-proof and pragmatic to simply go with JSON -- if only so that future readers of the code don't have to worry about learning about a new serialisation format. It's not a big deal at this point, but it might become one if features are added to serialisation format; that's what I'm saying.