partiality_threshold=0.4 in cosym, symmetry and ttr

[rjgildea]

Set the default partiality_threshold to 0.4 in dials.cosym, dials.symmetry and dials.two_theta_refine. The previous default of 0.99 could occasionally be overzealous in rejecting reflections, especially with particularly narrow wedges. This issue was identified by errors encountered running xia2.multiplex on James Holton's "Micro-focus Data Processing Challenge" dataset.

[codecov]

Codecov Report

Merging #1470 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1470   +/-   ##
=======================================
  Coverage   65.81%   65.82%           
=======================================
  Files         614      614           
  Lines       69410    69410           
  Branches     9508     9508           
=======================================
+ Hits        45685    45691    +6     
+ Misses      21887    21884    -3     
+ Partials     1838     1835    -3     

[dagewa]

For dials.two_theta_refine is the idea to use the same set of reflections as used in dials.cosym and dials.symmetry (for data sets that are scaled)? For the latter two programs, this change increases the number of reflections accepted, whereas for dials.two_theta_refine it decreases the number, but as a result essentially the same set of reflections gets used. Is that right?

[rjgildea]

for dials.two_theta_refine it decreases the number

Perhaps I'm missing something, but I don't understand why this would be the case? The default partiality_threshold for dials.util.filter_reflections.filter_reflection_table is 0.99, and previously this parameter was essentially hard-coded in dials.two_theta_refine and not user-controllable:

https://github.com/dials/dials/blob/master/util/filter_reflections.py#L457

[rjgildea]

With module load dials/latest:

$ dials.two_theta_refine $(dials.data get -q l_cysteine_4_sweeps_scaled)/scaled_20_25.{expt,refl} | egrep 'Removed|Summary'
Removed 20 reflections below partiality threshold
Summary statistics for 3585 observations matched to predictions:

With this PR:

$ dials.two_theta_refine $(dials.data get -q l_cysteine_4_sweeps_scaled)/scaled_20_25.{expt,refl} | egrep 'Removed|Summary'
Summary statistics for 3605 observations matched to predictions:

I.e. with this PR, fewer reflections are removed (none in this case) for being below the partiality threshold, and more reflections in total are used in refinement.

[dagewa]

I see, thanks. I didn't realise anything was filtered previously in dials.two_theta_refine - I thought it operated on all the reflections it was given. I didn't notice that it called filter_reflection_table already, I just assumed that was added here. All clear 👍

[dagewa]

It makes sense that these tools use the same threshold for post-scaling reflection selection. I did wonder whether it would make sense for filter_reflection_table to set a flag, but I think that can be a separate discussion.