Beta lactamase

Processing in Detail with Dui2

Introduction

This tutorial follows the same steps as this command line tutorial , but here processing will be driven graphically through the DIALS User Interface, Dui2.

Tutorial data

The following example uses a Beta-lactamase dataset collected using beamline I04 at Diamond Light Source, and reprocessed especially for these tutorials.

[Hint:] If you are physically at Diamond on the CCP4 Workshop, then this data is already available in your training data area. After typing module load ccp4-workshop you'll be moved to a working folder, with the data already located in the tutorial-data/summed sub directory.

The data is otherwise available for download from here. We'll only be using the first run of data in this tutorial,C2sum_1.tar, extracted to a tutorial-data/summed subdirectory.

Import

When DUI starts up you will initially be presented by a window like the following.

At this stage you can resize various parts of the interface until you are comfortable with the layout, but you can’t do much else until the dataset is imported by DIALS. To do this, click on the "Select file(s)" button, navigate to the location of the images and select any one of them. Dui2 will automatically determine the filename template and will show that with a wildcard in the text box. If there are problems with this template it is possible to edit this before importing the images. Otherwise, just click the “Run” button to proceed.

What happens then is that that the metadata are read for all the images in the dataset. If these are consistent, then the dataset is imported and initial models for the “Beam”, “Scan” and “Detector” are created. The images are now also displayed within the “Image” tab. You can adjust the contrast and colour scheme by controls under the “Display info ...” pull-down.

Find Spots

The first “real” task in any processing using DIALS is the spot finding. To run this job, click on the “find spots” button at the upper-left of the window. When you do this you will see a new node will be created in the “History Tree”. This node is currently green, which indicates that it has not been run yet. By contrast, the "Import" step is blue, which means this has been run. In general, it is always possible to navigate between each step of processing by clicking on the relevant position in the history. Advanced users will find this gives a great deal of control, allowing them to keep track of complex history, including parallel branches.

Spot-finding, like most of the other processing steps in Dui2, presents user parameters at two levels of detail. The “Simple” tab contains the basic parameters that are the most commonly changed. The “Advanced” tab contains all of those again, plus other parameters that may be required for expert use with challenging data sets. In many cases, however, the default settings are fine.

Note that spot-finding is done on every image in the dataset. This means the job can take some time, but by default it will be run in parallel using multiple processors. To proceed, press the “Run” button below the input parameters.

Once the job is finished, the image viewer will display small blue boxes around the pixels that have been marked as strong. It is also useful to click on the “Report” tab and scroll down to the “Analysis of strong reflections”. This shows a graph of the number of strong spots found per image. In this case there is a pretty steady rate of around 150 spots found on each image. If instead we had seen the number of strong spots drop off over the dataset, or otherwise show large variability we would start to worry about issues such as radiation damage or a poorly-centred crystal.

The cyan button at the bottom left of the graph opens a help window with a description of how the appearance of this plot may be affected by various data collection issues. In the “Log Text” window you can see the text output from the dials.find_spots program, which also includes an ASCII-art version of this plot.

The default parameters for spot finding usually do a good job for Pilatus images, such as these. However they may not be optimal for data from other detector types, such as CCDs or image plates. If you have a case where spot-finding has gone badly, it may be helpful to debug using the dials.image_viewer and dials.reciprocal_lattice_viewer, which can be launched via buttons shown on the “Tools” tab.

In particular, the effect of changing the spot-finding parameters can be explored interactively with the dials.image_viewer. The image mode buttons at the bottom of the “Settings” window allow a preview of how the parameters affect the spot finding algorithm. The final image, (“threshold”) is the one on which spots were found, so ensuring this produces peaks at real diffraction spot positions will give the best chance of success.

The second external viewer, the dials.reciprocal_lattice_viewer, displays the strong spots in 3D, after mapping them from their detector positions to reciprocal space. In a favourable case you should be able to see the crystal’s reciprocal lattice by eye in the strong spot positions. Some practice may be needed in rotating the lattice to an orientation that shows off the periodicity in reciprocal lattice positions.

Although the reciprocal spacing is visible, in this data, there are clearly some systematic distortions. These will be solved during indexing.

Indexing

The next step will be indexing of the strong spots. Click on the “Index” button to move on to this step, and form a new node in the history tree. Here we see that the simple parameters allows to select between different “Indexing Methods”, the default of which is the 3D FFT algorithm. The other options include the 1D FFT (DPS) algorithm and a special version of the 3D FFT called real_space_grid_search, which is particularly useful for narrow wedges containing multiple lattices, but requires a known cell and space group to be set under the “Advanced” parameters. If we do know the cell and space group, these can also be set as hints for any of the other indexing algorithms. This can help in difficult cases and will be used to constrain the lattice during refinement. Otherwise indexing and refinement will be carried out in the primitive lattice using space group "P 1" .

In this case, keep the method set to the default "fft3d" and click “Run” to start the indexing job. Once the job has finished running, you can see in the “Report” tab if you expand the "Experimental geometry" field, that the experimental models have now been completed with a “Crystal” model.

Now let’s click through the rest of the tabs of output. First, on the “Image” tab you will now see that indexed strong spots are assigned Miller indices. By default only the nearest one to the mouse cursor is shown, but this can be changed under the “Display” settings. If you also click on the “Predictions” checkbox, under “Reflection Type” you will in addition see centroid positions and Miller indices for all predicted reflections, not just the strong spots.

Moving to the “Log” tab, it is worth reading through the output to understand what the indexing program has done. Inspecting the beginning of the log shows that the indexing step is done at a resolution lower than the full dataset; 1.84 Å:

Found max_cell: 94.4 Angstrom Setting d_min: 1.84 FFT gridding: (256,256,256)

The resolution limit of data that can be used in indexing is determined by the size of the 3D FFT grid, and the likely maximum cell dimension. Here we used the default 256³ grid points. These are used to make an initial estimate for the unit cell parameters.

What then follows are ‘macro-cycles’ of refinement where the experimental model is first tuned to get the best possible fit from the data, and then the resolution limit is reduced to cover more data than the previous cycle. 16 parameters of the diffraction geometry are tuned: 6 for the detector, one for beam angle, 3 crystal orientation angles and the 6 triclinic cell parameters. At each stage only 36000 reflections are used in the refinement job. In order to save time, a subset of the input reflections are used - by default using 100 reflections for every degree of the 360° scan.

We see that the first macrocycle of refinement makes a big improvement in the positional RMSDs:

  +--------+--------+----------+----------+------------+
  |   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |
  |        |        |     (mm) |     (mm) |      (deg) |
  |--------+--------+----------+----------+------------|
  |      0 |  36000 | 0.56197  | 0.55131  |    0.13413 |
  |      1 |  36000 | 0.24338  | 0.26503  |    0.15575 |
  |      2 |  36000 | 0.10648  | 0.13198  |    0.13476 |
  |      3 |  36000 | 0.055785 | 0.059567 |    0.10888 |
  |      4 |  36000 | 0.051066 | 0.052336 |    0.10514 |
  |      5 |  36000 | 0.050906 | 0.052372 |    0.10505 |
  |      6 |  36000 | 0.050901 | 0.052378 |    0.10505 |
  +--------+--------+----------+----------+------------+

Second and subsequent macrocycles are refined using the same number of reflections, but after extending to higher resolution. The RMSDs at the start of each cycle start off worse than at the end of the previous cycle, because the best fit model for lower resolution data is being applied to higher resolution reflections. As long as each macrocyle shows a reduction in RMSDs then refinement is doing its job of extending the applicability of the model out to a new resolution limit, until eventually the highest resolution strong spots have been included. The final macrocycle includes data out to 1.30 Å and produces a final model with RMSDs of 0.050 mm in X, 0.049 mm in Y and 0.104° in φ, corresponding to 0.29 pixels in X, 0.28 pixels in Y and 0.21 image widths in φ.

Despite the high quality of this data, we notice from the log that at each macrocycle there were some outliers identified and removed from refinement as resolution increases. Large outliers can dominate refinement using a least squares target, so it is important to be able to remove these. More about this is discussed below in Refinement. It’s also worth checking the total number of reflections that were unable to be assigned an index:

  +------------+-------------+---------------+-------------+
  |   Imageset |   # indexed |   # unindexed | % indexed   |
  |------------+-------------+---------------+-------------|
  |          0 |      107264 |           733 | 99.3%       |
  +------------+-------------+---------------+-------------+

because this can be an indication of poor data quality or a sign that more care needs to be taken in selecting the indexing parameters.

Now the “Report View” contains more information than just after spot-finding. The “Spot count per image” plot also contains information about the number of indexed spots. In addition there are heat maps giving information about the positions of indexed and unindexed spots. Here we see that most of the unindexed spots are found in the region around the rotation axis. The “Analysis of reflection centroids” plots provide lots of detail regarding how well the predicted spot positions match the observed positions, both in image space and as a function of the position within the rotation scan.

After indexing it can be useful to inspect the reciprocal lattice again under the “External Tools”. Now indexed/unindexed spots are differentiated by colour, and it is possible to see which spots were marked by dials.refine as outliers. If you have a dataset with multiple lattices present, it may be possible to spot them in the unindexed reflections.

In this case, we can see that the refinement has clearly resolved whatever systematic error was causing distortions in the reciprocal space view, and the determined reciprocal unit cell fits the data well:

YYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ