Skip to content

Capture processing history at the point of MTZ export#2924

Merged
dagewa merged 10 commits intomainfrom
history-as-experiment-object
Nov 5, 2025
Merged

Capture processing history at the point of MTZ export#2924
dagewa merged 10 commits intomainfrom
history-as-experiment-object

Conversation

@dagewa
Copy link
Copy Markdown
Member

@dagewa dagewa commented May 14, 2025

This is the DIALS companion to cctbx/dxtbx#816, and requires that to be merged first. Changes here:

  • History is explicitly copied from the imported experiments in dials.index, fixing the issue that this got lost because of the way dials.index constructs experiments
  • When dials.integrate and dials.scale write .expt files they flag that these lines are interesting for dials.export
  • dials.export then takes these interesting lines and adds appropriate MTZ history items

As an example, these lines are written to the MTZ history for standard single crystal rotation processing:

From dials.integrate v3.22.dev0, run on 2025-05-14 at 10:53:02 UTC
From dials.scale v3.22.dev0, run on 2025-05-14 at 10:53:18 UTC

The version string looks a bit odd. When I run dials.version on the same computer I see 3.dev.1314-g6a06be331. The way version information is extracted for experiment history is by using importlib.metadata.version, which for DIALS gives:

>>> importlib.metadata.version("dials")
'3.22.dev0'

@ndevenish is this importlib.metadata.version call for DIALS correct? For the history functionality in dxtbx I could not rely on dials.version, because I can't be sure the software writing the .expt is actually DIALS. Indeed for xia2.multiplex I would get '3.9.dev184+g8e8dffbc' as the version string here on my computer.

@dagewa
Copy link
Copy Markdown
Member Author

dagewa commented May 14, 2025

I note that for the release version of DIALS v3.22.1, importlib.metadata.version("dials") also gives '3.22.dev0'

@dagewa
Copy link
Copy Markdown
Member Author

dagewa commented Aug 12, 2025

Possibly rethinking this to use the MTZ appendix rather than MTZ history.

@dagewa dagewa marked this pull request as draft October 16, 2025 14:29
dagewa and others added 3 commits October 24, 2025 06:38
Revert to a single line in MTZ history, intended for dials.export to note itself as the MTZ creator. Meanwhile, just log the integration and scaling history. This is a placeholder for putting that information later into e.g. the MTZ Appendix
@dagewa dagewa marked this pull request as ready for review October 24, 2025 08:35
@dagewa dagewa changed the title Write integrate and scale history to MTZ Capture processing history at the point of MTZ export Oct 24, 2025
@dagewa
Copy link
Copy Markdown
Member Author

dagewa commented Oct 24, 2025

@yangha7 I changed this PR so that it does not write the integration and scale history to the MTZ history, but it just captures and logs these for later addition to the MTZ Appendix, if that's what we decide on.

I think we should merge this in preparation for whatever we decide to do as it has useful functionality such as changing the datestamps to UTC, fixing dials.index so it does not lose history, and ensure dials.integrate and dials.scale mark their history entries as integrated and scaled, so those lines can be easily extracted from the full processing history.

@dagewa dagewa merged commit db530b9 into main Nov 5, 2025
12 checks passed
@dagewa dagewa deleted the history-as-experiment-object branch November 5, 2025 10:28
ndevenish pushed a commit that referenced this pull request Dec 9, 2025
* Ensure history from import is copied into the indexed experiments
* Flag `integrated` and `scaled` in the history, so other programs can set this (i.e. not just `dials.integrate` and `dials.scale`)
* Change MTZ export history timestamp to use UTC only (not UTC _and_ local time zone)
* Capture integrate and scaled history lines to write... somewhere. Maybe MTZ appendix? TBC.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants