Parallelize the LoadLotsOfInstruments system test

[mantid-builder]

This issue was originally TRAC 7908

Original Reporter: Russell Taylor

This ticket is blocked by :

• TRAC6261

It's taking 30 minutes or more to run in places, and should be trivial to parallelize using the multiprocessing module. In general, we ought to prefer parallelizing the Mantid code itself rather than the tests, but the LoadInstrument itself is not and probably can't be parallelized.

[mantid-builder]

Russell Taylor (2013-09-10T18:23:47):
Re http://trac.mantidproject.org/mantid/ticket/7908. Move exception handling into loading function.

1625cec

---

Russell Taylor (2013-09-10T18:23:48):
Re http://trac.mantidproject.org/mantid/ticket/7908. Parallelize LoadLotsOfInstruments test.

Uses the multiprocessing module to run LoadEmptyInstrument on the different
IDF files in parallel.

3032a3c

---

Russell Taylor (2013-09-10T18:58:44):
Something deep within OpenCascade concerning the writing of the vtp files is scuppering me here - it's locking up for some reason.

Re-visit this after http://trac.mantidproject.org/mantid/ticket/6261 is done. In fact, that will likely remove most of the benefit of this ticket anyway as most of the time is in generating these files. Here are the timings on my machine:

• Parallel, not writing vtp files: 21s
• Not parallel, vtp files there: 95s
• Not parallel, writing vtp files: 20 minutes

  ---

Russell Taylor (2013-09-11T00:39:02):
The parallel code works fine if all the vtp files are already there.

---

Russell Taylor (2013-10-15T17:34:38):
The blocking ticket has been moved out of the 3.0 milestone so this one has to be as well.

---

Russell Taylor (2014-05-19T19:37:11):
Re http://trac.mantidproject.org/mantid/ticket/7908. Move exception handling into loading function.

af9b036

---

Russell Taylor (2014-05-19T19:37:12):
Re http://trac.mantidproject.org/mantid/ticket/7908. Parallelize LoadLotsOfInstruments test.

Uses the multiprocessing module to run LoadEmptyInstrument on the different
IDF files in parallel.

208e262

---

Russell Taylor (2014-05-19T19:55:40):
The above 2 commits are a rebasing of the previous 2 onto a more recent snapshot of master.

---

Russell Taylor (2014-06-17T15:26:50):
The work is done and the branch is at https://github.com/mantidproject/systemtests/tree/feature/7908_parallelize_loadlotsofinstruments

This just needs the blocking ticket to be done before it can be merged and run without hanging.

---

@NickDraper (2014-12-08T10:25:39):
Moved to the backlog at the code freeze of R3.3

[martyngigg]

Not high priority enough to push forward with.

Archiving for a rainy day.