Better support of pseudo-merohedral twinning

[biochem-fan]

Processing a chemical crystallography dataset, I encountered pseudo-merohedral twinning, where low-to-mid resolution spots were overlapped while higher resolution spots were clearly separated. Processing this as a single lattice dataset and running twin refinement, the structure was solved, but I saw "ghost" densities. Another sweep of the same crystal collected at a different $\chi$ angle gave clean density by twin refinement. In this second sweep, the high resolution spots were not separated (due to the way the crystal was oriented in the beam). My interpretation is that the (apparent) twin fraction of integrated intensities from the first sweep depended on the resolution, because the degree of spot overlaps varied.

Indexing of the first sweep with max_lattice=2 didn't work, because almost all low resolution reflections were assigned to the first crystal orientation and only high resolution spots were left for the second round of indexing (see a dials.rlv screenshot below).

SHELX supports refinement of such datasets with HKL5 format:

(b) In cases where only some reflections have contributions from more than one domain (nonmerohedral twins or twinning by reticular merohedry) the .hkl file must be edited and the index transformations applied to individual contributors, which are also assigned component numbers. The code HKLF 5 is used to read in this file; no TWIN command should be used.
http://www.csb.yale.edu/userguides/datamanip/shelx/97/ch_6.pdf

A less well-known program from Phil Evans, called FECKLESS, was developed in this direction but as far as I know, never fully completed.

I know this is a rare use-case in MX but is probably an important case for CX. Ideally, DIALS should be able to

1. index overlapping lattices and
2. export into the HKLF5 format

If this is too niche to be supported, what is the best strategy within the current capacity of the software?

[dagewa]

Linking this to #1890 and #2296