minor changes for reprocessing in STHML

[XeBoris]

Just minor changes for Stockholm processing. (More CPUs and another Anaconda locational)

[tunnell]

I don't think you want 40 cores. @JelleAalbers can you comment? I thought pax got unstable past 20 and you'll also hit I/O issues. @XeBoris have you done the test where you see that 40 is better than 20 or 10?

[XeBoris]

The question for 40 cores is arbitrary here. I just put in a high number because a single login node has up to 48 cores here in Stockholm. I just checked some log files and couldn't find anything unexpected. I didn't know that pax becomes unstable beyond 20 cores.
But we can also go with 20. I do not have measurement how was the PDC processing.

[tunnell]

I thought we saw crashes at some point. It may not be the case anymore. It's just worth checking.

[JelleAalbers]

Pax shouldn't be unstable anymore with high number of cores, this was the case when we didn't have proper file lock protection of the root class compilation.

Reading and writing are not parallelized, so once either of these becomes the bottleneck there's no point in parallelizing further. At what number of cores that occurs depends on your system hardware and the data you're processing; you can find out by processing a hundred events or so with one core so you get a timing report. For example, on an event builder machine processing a recent dataset (160923_0018), I get:

+----------------------------+-------+-------------+------+-----------+
| Plugin                     |     % | /event (ms) |  #/s | Total (s) |
+----------------------------+-------+-------------+------+-----------+
| ReadZipped                 |   0.6 |        18.4 | 54.2 |       1.9 |
| DecodeZPickle              |   5.5 |       168.9 |  5.9 |      17.1 |
| SortPulses                 |   0.7 |        22.4 | 44.7 |       2.3 |
| ConcatenateAdjacentPulses  |   0.5 |        15.0 | 66.7 |       1.5 |
| CheckBounds                |   0.5 |        16.4 | 61.0 |       1.7 |
| FindHits                   |  16.0 |       492.9 |  2.0 |      49.8 |
| GapSizeClustering          |   2.2 |        67.5 | 14.8 |       6.8 |
| RejectNoiseHits            |   1.4 |        42.9 | 23.3 |       4.3 |
| SumWaveform                |  20.3 |       625.4 |  1.6 |      63.2 |
| LocalMinimumClustering     |   7.7 |       236.9 |  4.2 |      23.9 |
| NaturalBreaksClustering    |   5.2 |       161.3 |  6.2 |      16.3 |
| BasicProperties            |   2.3 |        70.5 | 14.2 |       7.1 |
| SumWaveformProperties      |   2.7 |        84.6 | 11.8 |       8.5 |
| CountCoincidentNoisePulses |   0.4 |        13.2 | 75.6 |       1.3 |
| PosRecWeightedSum          |   0.4 |        12.8 | 78.3 |       1.3 |
| PosRecMaxPMT               |   0.2 |         5.9 |      |       0.6 |
| PosRecRobustWeightedMean   |   1.5 |        44.7 | 22.4 |       4.5 |
| PosRecNeuralNet            |   1.1 |        33.2 | 30.1 |       3.4 |
| PosRecTopPatternFit        |  21.1 |       648.4 |  1.5 |      65.5 |
| HitpatternSpread           |   1.9 |        57.1 | 17.5 |       5.8 |
| AdHocClassification1T      |   0.0 |         0.7 |      |       0.1 |
| BuildInteractions          |   0.0 |         0.6 |      |       0.1 |
| BasicInteractionProperties |   0.1 |         3.3 |      |       0.3 |
| DeleteLowLevelInfo         |   1.9 |        58.1 | 17.2 |       5.9 |
| EncodeROOTClass            |   5.0 |       154.3 |  6.5 |      15.6 |
| WriteROOTClass             |   0.7 |        22.4 | 44.7 |       2.3 |
| TOTAL                      | 100.0 |      3077.9 |  0.3 |     310.9 |
+----------------------------+-------+-------------+------+-----------+

Since reading (first plugin) and writing (last plugin) each take less than 1% of the time, we could easily go to as much cores as the machine has (40).

Of course there are other overheads as well that we do not quantify. The only way to know if you're still getting improvements when you raise the number of cores is to measure and compare the performance.

[XeBoris]

@JelleAalbers Thank you for clarifying this point.
@tunnell Another error came from the fact that the created shared libraries in ~/.local/rootpy/dict/... are not deleted by pax. This is in general not an issue of pax. In my case, the PDC home directory is limited to 5 GB which ran out of storage at some point. I just deleted the files and started again. I already asked Jelle about this but I haven't had time to go into the details of the problem.