-
Notifications
You must be signed in to change notification settings - Fork 47
Performance Week
Graeme Winter edited this page Sep 13, 2019
·
5 revisions
Following obvious things to look at:
- dials.export format=mtz
- spot finding from still images and rotation data
- integration
- importing / wrangling stills - do we want one image set per experiment?
Justification:
- hits us badly for fine sliced data - very badly. If you have 36,000 x 0.01 degrees (which is extreme but possible) then simply exporting MTZ file takes a substantial amount of the overall wall-clock time
- everything we do involves finding spots. From still images has become broken compared to 1.14 series because of how the iteration through image sets works - from rotation sets we just want to be faster if possible
- integration would obviously be nice but …
- how we handle stills after we conceptually retired image sets remains to be an issue
2, 4 will be blockers for rolling out DIALS 2.0 for VMXi, 1 should be a relatively easy fix
See also: https://github.com/dials/dials/projects/13 and https://github.com/dials/dials/wiki/Large-datasets
Test cases: Absurd fine slicing: /dls/mx-scratch/dials/fine-sliced-insulin -
Timing report:
0.0s 1. ◯ dials.generate_mask (4.3s)
4.3s 2. ◯ dials.find_spots (30.6m)
1840.6s 3. ◯ dials.detect_blanks (2.6s)
1843.2s 4. ◯ dials.search_beam_position (11.6s)
1854.9s 5. ◯ dials.index (35.9s)
1892.7s 6. ◯ dials.refine_bravais_settings (44.4s)
1940.9s 7. ◯ dials.reindex (1.9s)
1943.3s 8. ◯ dials.reindex (2.5s)
1946.2s 9. ◯ dials.refine (11.7s)
1957.9s 10. ◯ dials.refine (32.2s)
1993.3s 11. ◯ dials.integrate (40.3m)
4412.2s 12. ◯ dials.report (24.0s)
4438.2s 13. ◯ dials.export (18.2m)
5533.3s 14. ◯ dials.symmetry (23.2s)
5556.5s 15. ◯ dials.reindex (5.5s)
5563.6s 16. ◯ dials.scale (17.0m)
6585.7s 17. ◯ xia2.resolutionizer (3.4s)
6589.1s 18. ◯ dials.scale (17.1m)
7615.0s 19. ◯ dials.space_group (4.8s)
7620.6s 20. ◯ dials.export (15.1m)
8524.2s 21. ◯ dials.merge (4.3s)
8532.0s 22. ◯ dials.two_theta_refine (4.2s)
8536.6s 23. ◯ ctruncate (0.6s)
8537.3s 24. ◯ freerflag (0.3s)
8537.5s 25. ◯ freerflag (0.3s)
8543.0s 26. ◯ xia2.report (11.8s)
This was run on a 40-core Hamilton node. Suspect that most of the time in dials.scale is actually spent exporting MTZ files 🙄
Also "bog standard" data sets are useful: e.g. /dls/i03/data/2019/cm23003-4/20190911/SmarGon/rotation_calibration1/th1_Oo_Cc_Pp_1_master.h5