Metrology refinement

After initial indexing we can refine the detector metrology.

cd $scratch
mkdir metrology; cd metrology; mkdir t000_refine; cd t000_refine
cspad.cbf_metrology $scratch/results/r001*/000/out $scratch/results/r002[0-2]/000/out flat_refinement=True flat_refinement_with_distance=True method=expanding n_subset=3000 n_subset_method=n_refl n_refl_panel_list=10,26,42,58 | tee out.log

This will take quite some time. It reads all of the indexed data and finds the images with the most reflections on the corners so the best metrology can be found. Alternatively, the user can use a single run, or pick random images by leaving out n_subset_method and n_refl_panel_list.

Cycles of re-indexing and metrology refinement can be done in two ways. A) Re-index the data from scratch using the new metrology, then re-refine the detector. B) Just re-index the same set of images used in the previous cycle. Option B was used for Brewster 2018, but both options are described here.

Option A: re-index from scratch

After the first round of metrology refinement, a 0-end.data.cspad is produced. Deploy this file to $calib/geometry as before (using a soft link), then re-index the data as done in initial indexing. Then, re-refine the metrology using the above command. Repeat deploying the new metrology, re-indexing, and re-refinement until convergence.

Option B: re-index small subset

During metrology refinement, the file cspad_combined_experiments.json is produced, which includes links to every cbf file used. To re-index only those 3000 cbfs, first create a directory with soft links to those cbfs:

cd $scratch/metrology/t000_refine
mkdir data; cd data
for i in `grep cbf ../cspad_combined_experiments.json | sed 's/\"//g'`; do ln -fns $i .; done

Next, set up the re-indexing folder and phil

cd ..; mkdir v2_reindex; cd v2_reindex

with the phil file comprising these lines:

input.reference_geometry=$scratch/metrology/t000_refine/cspad_refined_experiments_step7_level2.json
spotfinder.lookup.mask = $scratch/masks/mask.pickle
spotfinder {
  filter {
    min_spot_size = 2
  }
  threshold {
    dispersion {
      gain = 25
      global_threshold = 100
    }
  }
}
indexing {
  known_symmetry {
    space_group = "P 61 2 2"
    unit_cell = 92.9 92.9 130.4 90 90 120
  }
  refinement_protocol {
    d_min_start = 1.7
  }
}

Use dials.stills_process to re-index the data:

bsub -n 36 -q psanaq -o log.out mpirun dials.stills_process ../data process.phil mp.method=mpi mp.nproc=36

Then re-refine the metrology:

cd ../; mkdir v3_refine; cd v3_refine
cspad.cbf_metrology ../v2_reindex flat_refinement=True flat_refinement_with_distance=True method=expanding

Repeat until convergence