Performance Week

Following obvious things to look at:

1. dials.export format=mtz
2. spot finding from still images and rotation data
3. integration
4. importing / wrangling stills - do we want one image set per experiment?

Justification:

1. hits us badly for fine sliced data - very badly. If you have 36,000 x 0.01 degrees (which is extreme but possible) then simply exporting MTZ file takes a substantial amount of the overall wall-clock time
2. everything we do involves finding spots. From still images has become broken compared to 1.14 series because of how the iteration through image sets works - from rotation sets we just want to be faster if possible
3. integration would obviously be nice but …
4. how we handle stills after we conceptually retired image sets remains to be an issue

2, 4 will be blockers for rolling out DIALS 2.0 for VMXi, 1 should be a relatively easy fix

See also: https://github.com/dials/dials/projects/13 and https://github.com/dials/dials/wiki/Large-datasets

Test cases: Absurd fine slicing: /dls/mx-scratch/dials/fine-sliced-insulin -

Timing report:
   0.0s   1. ◯     dials.generate_mask (4.3s)
   4.3s   2. ◯     dials.find_spots (30.6m)
1840.6s   3. ◯     dials.detect_blanks (2.6s)
1843.2s   4. ◯     dials.search_beam_position (11.6s)
1854.9s   5. ◯     dials.index (35.9s)
1892.7s   6. ◯     dials.refine_bravais_settings (44.4s)
1940.9s   7. ◯     dials.reindex (1.9s)
1943.3s   8. ◯     dials.reindex (2.5s)
1946.2s   9. ◯     dials.refine (11.7s)
1957.9s  10. ◯     dials.refine (32.2s)
1993.3s  11. ◯     dials.integrate (40.3m)
4412.2s  12. ◯     dials.report (24.0s)
4438.2s  13. ◯     dials.export (18.2m)
5533.3s  14. ◯     dials.symmetry (23.2s)
5556.5s  15. ◯     dials.reindex (5.5s)
5563.6s  16. ◯     dials.scale (17.0m)
6585.7s  17. ◯     xia2.resolutionizer (3.4s)
6589.1s  18. ◯     dials.scale (17.1m)
7615.0s  19. ◯     dials.space_group (4.8s)
7620.6s  20. ◯     dials.export (15.1m)
8524.2s  21. ◯     dials.merge (4.3s)
8532.0s  22. ◯     dials.two_theta_refine (4.2s)
8536.6s  23. ◯     ctruncate (0.6s)
8537.3s  24. ◯     freerflag (0.3s)
8537.5s  25. ◯     freerflag (0.3s)
8543.0s  26. ◯     xia2.report (11.8s)

This was run on a 40-core Hamilton node. Suspect that most of the time in dials.scale is actually spent exporting MTZ files 🙄

Also "bog standard" data sets are useful: e.g. /dls/i03/data/2019/cm23003-4/20190911/SmarGon/rotation_calibration1/th1_Oo_Cc_Pp_1_master.h5