More general failure criteria

[glwagner]

We currently identified "failed" particles by checking if the forward map contains NaN:

ParameterEstimocean.jl/src/EnsembleKalmanInversions.jl

Lines 281 to 285 in 11e197d

	# Handle failed particles
	nan_values = column_has_nan(Gⁿ)
	failed_columns = findall(nan_values) # indices of columns (particles) with `NaN`s
	successful_columns = findall(.!nan_values)
	some_failures = length(failed_columns) > 0

ParameterEstimocean.jl/src/resampling.jl

Line 43 in 11e197d

column_has_nan(G) = vec(mapslices(any, isnan.(G); dims=1))

However, this failure criterion is too restrictive. For example, a forward map can "effectively fail" if it contains values with vastly different magnitudes than other particles. In that case, the EKI update will fail because the correlation matrices cannot be calculated within double precision.

To handle this problem and also perhaps to increase the quality of the EKI algorithm, I think we should support more general failure criteria than "forward map contains NaN". One generalization is to mark a column as failed if its norm is much greater (say by 1e9) than the median forward map. Other criteria could also be developed (perhaps also depending on the raw simulation output, rather than the forward map output).