Multiple stills view

[dermen]

This fixes various bugs that manifest in the image viewer when viewing stills. In particular, combined experiments can now be parsed with reflections placed on the correct image. Further, when paging through multiple images, I noticed that the caching of the dispersion_debug_memo was seemingly broken for my use case, so I simply bypassed it for viewing stills. I think with these commits, the image viewer is functional the way I expect it should be for stills. Maybe its not the ideal way to merge things into master, but at least its a start.

In order to demo the various artifacts that you would experience on the master branch, I designed a script. This will generate a sacla HDF5 file with 5 images:

from __future__ import absolute_import, division, print_function
import numpy as np
from scipy import constants

from cctbx import sgtbx, miller
from cctbx.crystal import symmetry
from dxtbx.model.beam import BeamFactory
from dxtbx.model.crystal import CrystalFactory
from dxtbx.model.detector import DetectorFactory
from scitbx.array_family import flex
from scitbx.matrix import sqr, col
from simtbx.nanoBragg import nanoBragg, shapetype

print("Make a randomly oriented xtal")
# make a randomly oriented crystal..
nshots = 5
seeds = range(nshots)
imgs = []

for seed in seeds:

  np.random.seed(seed)
  # make random rotation about principle axes
  x = col((-1, 0, 0))
  y = col((0, -1, 0))
  z = col((0, 0, -1))
  rx, ry, rz = np.random.uniform(-180, 180, 3)
  RX = x.axis_and_angle_as_r3_rotation_matrix(rx, deg=True)
  RY = y.axis_and_angle_as_r3_rotation_matrix(ry, deg=True)
  RZ = z.axis_and_angle_as_r3_rotation_matrix(rz, deg=True)
  M = RX*RY*RZ
  real_a = M*col((79, 0, 0))
  real_b = M*col((0, 79, 0))
  real_c = M*col((0, 0, 38))
  # dxtbx crystal description
  cryst_descr = {'__id__': 'crystal',
                 'real_space_a': real_a.elems,
                 'real_space_b': real_b.elems,
                 'real_space_c': real_c.elems,
                 'space_group_hall_symbol': ' P 4nw 2abw'}

  print("Make a beam")
  # make a beam
  ENERGY = 9000
  ENERGY_CONV = 1e10*constants.c*constants.h / constants.electron_volt
  WAVELEN = ENERGY_CONV/ENERGY
  # dxtbx beam model description
  beam_descr = {'direction': (0.0, 0.0, 1.0),
               'divergence': 0.0,
               'flux': 1e11,
               'polarization_fraction': 1.,
               'polarization_normal': (0.0, 1.0, 0.0),
               'sigma_divergence': 0.0,
               'transmission': 1.0,
               'wavelength': WAVELEN}

  # make a detector panel
  # monolithic camera description
  print("Make a dxtbx detector")
  detdist = 100.
  pixsize = 0.1
  im_shape = 1536, 1536
  det_descr = {'panels':
                 [{'fast_axis': (1.0, 0.0, 0.0),
                   'slow_axis': (0.0, -1.0, 0.0),
                   'gain': 1.0,
                   'identifier': '',
                   'image_size': im_shape,
                   'mask': [],
                   'material': '',
                   'mu': 0.0,
                   'name': 'Panel',
                   'origin': (-im_shape[0]*pixsize/2., im_shape[1]*pixsize/2., -detdist),
                   'pedestal': 0.0,
                   'pixel_size': (pixsize, pixsize),
                   'px_mm_strategy': {'type': 'SimplePxMmStrategy'},
                   'raw_image_offset': (0, 0),
                   'thickness': 0.0,
                   'trusted_range': (-1e7, 1e7),
                   'type': ''}]}

  # make the dxtbx objects
  BEAM = BeamFactory.from_dict(beam_descr)
  DETECTOR = DetectorFactory.from_dict(det_descr)
  CRYSTAL = CrystalFactory.from_dict(cryst_descr)

  # make a dummie HKL table with constant HKL intensity
  # this is just to make spots
  DEFAULT_F = 1e3
  symbol = CRYSTAL.get_space_group().info().type().lookup_symbol()  # this is just P43212
  sgi = sgtbx.space_group_info(symbol)
  symm = symmetry(unit_cell=CRYSTAL.get_unit_cell(), space_group_info=sgi)
  miller_set = symm.build_miller_set(anomalous_flag=True, d_min=1.6, d_max=999)
  Famp = flex.double(np.ones(len(miller_set.indices())) * DEFAULT_F)
  Famp = miller.array(miller_set=miller_set, data=Famp).set_observation_type_xray_amplitude()

  Ncells_abc = 20, 20, 20
  oversmaple = 2

  # do the simulation
  print("Do the initial simulation")
  SIM = nanoBragg(DETECTOR, BEAM, panel_id=0)
  SIM.flux=1e12
  SIM.spot_scale = 10000
  SIM.beamsize_mm=0.1
  SIM.Ncells_abc = Ncells_abc
  SIM.Fhkl = Famp
  SIM.Amatrix = sqr(CRYSTAL.get_A()).transpose()
  SIM.oversample = oversmaple
  SIM.xtal_shape = shapetype.Gauss
  bg = flex.vec2_double([(0,2.57),(0.0365,2.58),(0.07,2.8),(0.12,5),(0.162,8),(0.2,6.75),(0.18,7.32),(0.216,6.75),(0.236,6.5),(0.28,4.5),(0.3,4.3),(0.345,4.36),(0.436,3.77),(0.5,3.17)])
  SIM.Fbg_vs_stol = bg
  SIM.amorphous_sample_thick_mm = 0.1
  SIM.amorphous_density_gcm3 = 1
  SIM.amorphous_molecular_weight_Da = 18
  SIM.add_nanoBragg_spots()
  print("Add noise")
  SIM.adc_offset_adu = 10
  SIM.psf_fwhm = 0
  
  SIM.add_noise()

  # write the simulation to disk using cbf writer
  #cbf_filename = "test_shot_seed%d.cbf" % seed
  #print("write simulation to disk (%s)" % cbf_filename)
  #SIM.to_cbf(cbf_filename)

  img = SIM.raw_pixels.as_numpy_array()
  imgs.append(img)
  SIM.free_all()
  print("Done with image %d" % seed)

import h5py
with h5py.File("test_sacla.h5", "w") as h:
    for i_img, img in enumerate(imgs):
        h.create_dataset("run_0/detector_2d_assembled_1/tag_%d/detector_data" % i_img, data=img)
    h.create_dataset("file_info/run_number_list", data=[0])
    h.create_dataset("run_0/detector_2d_assembled_1/detector_info/pixel_size_in_micro_meter", data=[pixsize*1000, pixsize*1000])
    h.create_dataset("run_0/run_info/sacla_config/photon_energy_in_eV", data=ENERGY)
    tags = range(len(imgs))
    h.create_dataset("run_0/event_info/tag_number_list", data=list(tags))

After running this script you can process the file using stills_process:

dials.stills_process  test_sacla.h5 unit_cell="79 79 38 90 90 90" space_group=P43212  dispatch.integrate=False output.output_dir=test_shots_sacla

Navigate to the output folder, combine the experiments, and view the results

cd test_shots_sacla
dials.combined_experiments  *refined.expt *indexed.refl
dials.image_viewer combined.*

On master branch you will notice all the spots are overlaid in the first image. Further , if you click on threshold view, then page through images, you will notice the display getting stuck, not updating properly.

In this current branch (multiple_stills_view), these problems are fixed, and also the time to launch display of the combined.expt should be quicker.

[codecov]

Codecov Report

Merging #1463 (960a64e) into master (559e781) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #1463      +/-   ##
==========================================
- Coverage   66.67%   66.62%   -0.05%     
==========================================
  Files         616      616              
  Lines       69187    69228      +41     
  Branches     9553     9567      +14     
==========================================
- Hits        46127    46125       -2     
- Misses      21121    21163      +42     
- Partials     1939     1940       +1     

[Anthchirp]

Added a few comments

[Anthchirp]

I would strongly suggest against using force-pushing in pull requests, as it makes it impossible to see what has changed since I reviewed the code (I will not review all of it again). It also breaks up the conversation flow, as it now looks like all commits were made after the code coverage determination and the review comments. This is not what happened.

If you want to update the branch you should instead merge master into your branch.

[dermen]

@Anthchirp , yikes, yea that was not the ideal thing for me to do. The two commits Apply suggestions from code review and more suggestions are the direct responses to your review questions.

[phyy-nx]

@rjgildea can you review? We'd like to know if this is affecting sweeps/grid scans at all.