Initial noise convergence test implementation and outline noise-only simulated obs for testing

[e-koch]

Add additional convergence test based on sampling from the noise distribution.

The main thing this PR adds is check_noise_convergence. It's an optional, additional, convergence check that handles cases where a few clean iterations at the noise threshold are included in multiple imaging loops, leading to large fractional changes in the total flux despite being consistent with the noise. We've collectively started hitting chunked imaging cases where a noise-only channel chunk uses all 20 imaging loops and taking way longer to exit than should be needed.

The thinking behind this new test is:

• clean components represent the extremum of the residual image at each iteration.
• For n_pix finite pixels the extremum is drawn from both tails of the Gaussian, so Var(extremum) ≈ 2·ln(2·n_pix)·(gain·noise)^2.
• the sum S of N_total such extremal draws has Std(S) = sqrt(N_total) · gain · noise · sqrt(2·ln(2·n_pix)).
• we compute the z score = total_flux_iter / Std(S). The criterion passes (null hypothesis) if z_score < z_threshold.
• The threshold is set in run_imaging and loop_imaging from the image handler with kwarg convergence_noise_z_threshold. By default it is None, and this test is not included; passing any finite >0 value enables the noise convergence test, in addition to the existing total flux convergence.

We need the info dict from fullsummary=True in tclean and sdintimaging to know the number of iterations actually used, flux per major cycle, etc. The score above defaults to using the last 3 major cycles. There's a future option to correct by scale for multiscale cleaning as this info is also returned in the dictionary.

Trying this on a VLA HI test case for a cube with some low S/N detections, the noise Z-score criterion matches the total flux convergence (the 4 loops is the minimum hardcoded in the PHANGS recipe; otherwise both tests would pass after loop 2):

loopnum, deconvolver, niter, cycleniter, threshold, noise, model_flux, frac_delta_flux, noise_z_score
# column 1: Loop number.
# column 2: Deconvolver used in clean.
# column 3: Allocated number of iterations.
# column 4: Cycleniter used to force major cycles.
# column 5: Threshold supplied to clean.
# column 6: Noise level measured in residuals.
# column 7: Integrated model flux.
# column 8: Fractional change in flux from previous loop.
# column 9: Noise z-score.
0, multiscale, 100, 100, 0.02831184606846778Jy/beam, 0.007077961517116945Jy/beam, 1.3617980759704822Jy*chan, inf, 31.32496804221372
1, multiscale, 200, 200, 0.027029895700661062Jy/beam, 0.0067574739251652655Jy/beam, 1.8751552645808185Jy*chan, 0.37697012330149865, 9.468209524700818
2, multiscale, 400, 300, 0.025975869586093447Jy/beam, 0.006493967396523362Jy/beam, 1.8749897100265613Jy*chan, 8.828845130014407e-05, -0.005813716459525169
3, multiscale, 800, 400, 0.025743743418845406Jy/beam, 0.0064359358547113514Jy/beam, 1.8879628674496434Jy*chan, 0.006919055263987687, 0.6851589398047857

[e-koch]

Local version runs. However, my test case (snapshot VLA in C config) is biased to mean != 0 and isn't well suited as a test. Still working on a suitable noise-only test case. This is now corrected as the initial version of the z-score did not account for the number of pixels (draws) in the image in the expected variance.

[e-koch]

My local tests (full imaging case and the new unit tests) are passing

[e-koch]

The PR also includes additional tests that we can wrap into a broader testing suite:

• noise_only_ms_factory.py produces a very small ALMA 12m Band 6 simulated obs that is purely simulated noise
• minimal_key_writer.py sets up mock keys for the simulated noise-only obs (expandable for other test cases, too)
• 2 new test files for the noise convergence: one is unit tests for the check_noise_convergence w/o CASA; the other is a full pipeline run through staging and imaging with and without including the new convergence test. These all pass locally but more work is needed to get these and others working in the test suite.

[thomaswilliamsastro]

Just run through with this. I've pushed the fix for the "no clean components" case to this as well, and it all seems quite a bit faster -- it's running waaaay fewer major cycles. I used convergence_noise_z_threshold=2, but I'm not really sure what an appropriate value here is. Might be worth adding in a docstring or something?

I'll have a closer look through the code early next week, but this seems extremely promising!

[e-koch]

lower convergence_noise_z_threshold is less likely to trigger convergence. I'd say Z=2-3 is a reasonable choice that is unlikely to prematurely exit before convergence is reached. that lines up with the values I'm seeing (above) from some test cases.

That being said, we should absolutely test this to set a reasonable default.

[thomaswilliamsastro]

@e-koch Had a look through, there's a couple of comments I had. I think the last one is giving me the most confusion, but maybe I'm being dumb here

I've also done a quick check on the cubes with/without the criteria, and they look identical to me

[e-koch]

@thomaswilliamsastro addressed your comments. Should be good to merge now

[thomaswilliamsastro]

sorry, realised that I hadn't actually finished the review so this comment hadn't appeared. @e-koch it's just some clearup on "which pix" that I don't have clear in my head