dials.scale not coping well with errors

[Anthchirp]

am running
dials.scale '12_scaled.expt' '12_scaled.refl' 'nproc=4' 'model=physical' 'physical.decay_correction=True' 'physical.absorption_correction=False' 'physical.decay_interval=1.5' 'physical.scale_interval=1' 'full_matrix=False' 'error_model=basic' 'outlier_rejection=standard' 'reflection_selection.Isigma_range=2.000000,100000.000000' 'cut_data.d_min=6.65' 'output.unmerged_mtz=14_scaled_unmerged.mtz' 'output.merged_mtz=14_scaled.mtz' 'output.html=14_scaling.html' 'output.experiments=14_scaled.expt' 'output.reflections=14_scaled.refl'

on cctbx@2fe0518e6f30b140f9db55a1360076777ec7d7fd, DIALS@ afd9f7c, dxtbx@86f9d42a980c79e67908baa44a110577052612c5, xia2@a130679e517b2682bab9ee83dc3fbf6f2ddf4ee3.

Output:

3 outliers have been identified. 

Insufficient reflections (248 < 250) to perform error modelling.
Traceback (most recent call last):
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/scaler.py", line 218, in perform_error_optimisation
    self.params.weighting.error_model, Ih_table
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/error_model/engine.py", line 36, in run_error_model_refinement
    model = model_class(Ih_table.blocked_data_list[0], scope)
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/error_model/error_model.py", line 364, in __init__
    Ih_table, basic_params, min_partiality
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/error_model/error_model.py", line 409, in filter_unsuitable_reflections
    min_reflections_required=cls.min_reflections_required,
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/error_model/error_model.py", line 564, in filter_unsuitable_reflections
    % (n, min_reflections_required)
ValueError: Insufficient reflections (248 < 250) to perform error modelling.
Traceback (most recent call last):
  File "/scratch/wra62962/files/dials/build/../modules/dials/command_line/scale.py", line 258, in <module>
    run()
  File "/scratch/wra62962/files/dials/build/../modules/dials/command_line/scale.py", line 232, in run
    scaled_experiments, joint_table = run_scaling(params, experiments, reflections)
  File "/scratch/wra62962/files/dials/build/../modules/dials/command_line/scale.py", line 194, in run_scaling
    algorithm.run()
  File "/scratch/wra62962/files/dials/modules/dials/util/observer.py", line 45, in notify
    r = method(self, *args, **kwargs)
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/algorithm.py", line 208, in run
    self.scale()
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/algorithm.py", line 243, in scale
    self.scaler = scaling_algorithm(self.scaler)
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/algorithm.py", line 571, in scaling_algorithm
    do_error_analysis(scaler, reselect=False)
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/algorithm.py", line 514, in do_error_analysis
    scaler.perform_error_optimisation()
  File "/scratch/wra62962/files/dials/modules/dials/util/observer.py", line 46, in notify
    self.notify(event)
  File "/scratch/wra62962/files/dials/modules/dials/util/observer.py", line 68, in notify
    callback(self)
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/observers.py", line 376, in update
    scaler.error_model.clear_Ih_table()
  File "/scratch/wra62962/files/dials/modules/dials/algorithms/scaling/error_model/error_model.py", line 448, in clear_Ih_table
    del self.binner.Ih_table
AttributeError: Please report this error to dials-support@lists.sourceforge.net: Ih_table

it first crashes with a ValueError in

dials/algorithms/scaling/error_model/error_model.py

Lines 560 to 564 in afd9f7c

	if n < min_reflections_required:
	raise ValueError(
	"Insufficient reflections (%s < %s) to perform error modelling."
	% (n, min_reflections_required)
	)

which is caught and reported in

dials/algorithms/scaling/scaler.py

Lines 216 to 221 in afd9f7c

	try:
	model = run_error_model_refinement(
	self.params.weighting.error_model, Ih_table
	)
	except (ValueError, RuntimeError) as e:
	logger.info(e, exc_info=True)

but then for some reason we end up in

dials/algorithms/scaling/error_model/error_model.py

Lines 445 to 447 in afd9f7c

	def clear_Ih_table(self):
	"""Delete the Ih_table, to free memory."""
	del self.binner.Ih_table

where we are trying to delete something that wasn't set up, which then crashes with the uncaught second traceback.

Apart from the obvious issue for me this does raise the question whether an explicit deletion is required at all. I mean - do we need more memory than, say, integration? Or is the object layout such that we work around the garbage collector so that we need to track references manually?

Discovered via failing xia2 test:

pytest Test/regression/test_multiplex.py::test_proteinase_k_filter_deltacchalf --regression-full

[Anthchirp]

as far as I remember, this table object can get very large, with several entries for most reflections. References to the error model object are kept until the end so I think it is correct to delete this (if it's there of course)

Although achieving a legitimate goal, looking at this again I believe the object model needs a bit of an overhaul. Reaching into another object and deleting a public name is quite intrusive and prone to cause issues downstream. I believe it would be more sensible to rewrite this to use a .delete_table() function that then locks out other functions depending on the deleted data structure. Also should check whether Ih_table should be exposed as a public name at all.