CLI code overhaul.

The command-line arguments for hyperdrive subcommands are now much more
consistent. This is because they are using the same code internally. This meant
that, unfortunately, many lines of code were altered.

The preamble printed from subcommands is more consistent, prettier and
consolidates warnings.

Most hyperdrive subcommands now support argument files.

All input-vis-data reading subcommands can now average the visibilities before
doing work on them.

Progress bars and the model device are consolidated into "global variables",
allowing internal APIs to be simpler.

Channel indices are now generally represented by u16 rather than usize. This
hopefully helps performance but also prevents ridiculously large channel counts
from being considered.

Inspiration and insight was taken from
https://rust-cli-recommendations.sunshowers.io/index.html

CHANGELOG.md

@@ -6,8 +6,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic
 Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.2.2] - Unreleased
+## [0.3.0] - Unreleased
 ### Added
+- Support for averaging incoming visibilities in time and frequency *before*
+  doing any work on them.
+- When writing out visibilities, it is now possible to write out the smallest
+  contiguous band of unflagged channels.
 - Plots can be written to a specific directory, not only the CWD. Fixes #18.
 - Support for visibilities using the ant2-ant1 ordering rather than ant1-ant2.
 - Add new errors
@@ -18,6 +22,10 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 - Benchmarks
   - Raw MWA, uvfits and measurement set reading.
   - More CUDA benchmarks for the modelling code.
+- Support for "argument files". This is an advanced feature that most users
+  probably should avoid. Previously, argument files were supported for the
+  di-calibrate subcommand, but now it is more consistently supported among the
+  "big" subcommands.
 
 ### Fixed
 - When raw MWA data is missing gpubox files in what is otherwise a contiguous
@@ -34,9 +42,16 @@ Versioning](https://semver.org/spec/v2.0.0.html).
   even with the explicit "ignore input data tile flags".
 - Some aspects of hyperdrive weren't using user-specified array positions
   correctly. The help text also indicated the wrong units.
+- Fine-channel flags and fine-channel-per-coarse channel flags are now checked
+  for validity.
 
 ### Changed
 - The performance of CPU visibility modelling has been dramatically improved.
+- The command-line interface has been overhauled. Some things may be different,
+  but generally the options and flags are much more consistent between
+  subcommands.
+- The preamble to "big" subcommands, like di-calibrate, has been overhauled to
+  be much easier to read.
 - Plotting
   - Legend labels have changed to $g_x$, $D_x$, $D_y$, $g_y$ ($g$ for gain, $D$
     for leakage). Thanks Jack Line.

Cargo.toml

@@ -113,7 +113,7 @@ approx = "0.5.1"
 assert_cmd = "2.0.0"
 criterion = { version = "0.4.0", default_features = false }
 indoc = "2.0.1"
-marlu = { version = "0.10.0", features = ["approx"] }
+marlu = { version = "0.10.1", features = ["approx"] }
 ndarray = { version = "0.15.4", features = ["approx-0_5"] }
 serial_test = "2.0.0"
 tar = "0.4.38"

benches/bench.rs

@@ -18,16 +18,14 @@ use tempfile::Builder;
 use vec1::{vec1, Vec1};
 
 use mwa_hyperdrive::{
-    averaging::{Chanblock, Timeblock},
-    beam::{create_fee_beam_object, Delays},
-    di_calibrate::calibrate_timeblocks,
-    model,
+    calibrate_timeblocks, create_fee_beam_object,
+    model::{self, SkyModeller},
     srclist::{
         get_instrumental_flux_densities, ComponentType, FluxDensity, FluxDensityType,
         ShapeletCoeff, Source, SourceComponent, SourceList,
     },
-    CrossData, MsReader, Polarisations, RawDataCorrections, RawDataReader, TileBaselineFlags,
-    UvfitsReader,
+    Chanblock, CrossData, Delays, MsReader, Polarisations, RawDataCorrections, RawDataReader,
+    TileBaselineFlags, Timeblock, UvfitsReader,
 };
 
 fn model_benchmarks(c: &mut Criterion) {
@@ -73,9 +71,7 @@ fn model_benchmarks(c: &mut Criterion) {
                         },
                     );
                 }
-                let modeller = model::new_sky_modeller(
-                    #[cfg(feature = "cuda")]
-                    true,
+                let modeller = model::SkyModellerCpu::new(
                     &*beam,
                     &source_list,
                     Polarisations::default(),
@@ -87,8 +83,7 @@ fn model_benchmarks(c: &mut Criterion) {
                     MWA_LAT_RAD,
                     dut1,
                     apply_precession,
-                )
-                .unwrap();
+                );
 
                 b.iter(|| {
                     modeller
@@ -137,8 +132,7 @@ fn model_benchmarks(c: &mut Criterion) {
                         },
                     );
                 }
-                let modeller = model::new_sky_modeller(
-                    false,
+                let modeller = model::SkyModellerCuda::new(
                     &*beam,
                     &source_list,
                     Polarisations::default(),
@@ -196,9 +190,7 @@ fn model_benchmarks(c: &mut Criterion) {
                 },
             );
         }
-        let modeller = model::new_sky_modeller(
-            #[cfg(feature = "cuda")]
-            true,
+        let modeller = model::SkyModellerCpu::new(
             &*beam,
             &source_list,
             Polarisations::default(),
@@ -210,8 +202,7 @@ fn model_benchmarks(c: &mut Criterion) {
             MWA_LAT_RAD,
             dut1,
             apply_precession,
-        )
-        .unwrap();
+        );
 
         b.iter(|| {
             modeller
@@ -262,8 +253,7 @@ fn model_benchmarks(c: &mut Criterion) {
                         },
                     );
                 }
-                let modeller = model::new_sky_modeller(
-                    false,
+                let modeller = model::SkyModellerCuda::new(
                     &*beam,
                     &source_list,
                     Polarisations::default(),
@@ -332,9 +322,7 @@ fn model_benchmarks(c: &mut Criterion) {
                     },
                 );
             }
-            let modeller = model::new_sky_modeller(
-                #[cfg(feature = "cuda")]
-                true,
+            let modeller = model::SkyModellerCpu::new(
                 &*beam,
                 &source_list,
                 Polarisations::default(),
@@ -346,8 +334,7 @@ fn model_benchmarks(c: &mut Criterion) {
                 MWA_LAT_RAD,
                 dut1,
                 apply_precession,
-            )
-            .unwrap();
+            );
 
             b.iter(|| {
                 modeller
@@ -408,8 +395,7 @@ fn model_benchmarks(c: &mut Criterion) {
                         },
                     );
                 }
-                let modeller = model::new_sky_modeller(
-                    false,
+                let modeller = model::SkyModellerCuda::new(
                     &*beam,
                     &source_list,
                     Polarisations::default(),
@@ -443,16 +429,16 @@ fn calibrate_benchmarks(c: &mut Criterion) {
         index: 0,
         range: 0..num_timesteps,
         timestamps: vec1![
-            Epoch::from_gpst_seconds(1090008640.0),
-            Epoch::from_gpst_seconds(1090008641.0),
-            Epoch::from_gpst_seconds(1090008642.0),
-            Epoch::from_gpst_seconds(1090008643.0),
-            Epoch::from_gpst_seconds(1090008644.0),
-            Epoch::from_gpst_seconds(1090008645.0),
-            Epoch::from_gpst_seconds(1090008646.0),
-            Epoch::from_gpst_seconds(1090008647.0),
-            Epoch::from_gpst_seconds(1090008648.0),
-            Epoch::from_gpst_seconds(1090008649.0),
+            (Epoch::from_gpst_seconds(1090008640.0), 0),
+            (Epoch::from_gpst_seconds(1090008641.0), 1),
+            (Epoch::from_gpst_seconds(1090008642.0), 2),
+            (Epoch::from_gpst_seconds(1090008643.0), 3),
+            (Epoch::from_gpst_seconds(1090008644.0), 4),
+            (Epoch::from_gpst_seconds(1090008645.0), 5),
+            (Epoch::from_gpst_seconds(1090008646.0), 6),
+            (Epoch::from_gpst_seconds(1090008647.0), 7),
+            (Epoch::from_gpst_seconds(1090008648.0), 8),
+            (Epoch::from_gpst_seconds(1090008649.0), 9),
         ],
         median: Epoch::from_gpst_seconds(1090008644.5),
     });
@@ -464,7 +450,7 @@ fn calibrate_benchmarks(c: &mut Criterion) {
         chanblocks.push(Chanblock {
             chanblock_index: i_chanblock as _,
             unflagged_index: i_chanblock as _,
-            _freq: 150e6 + i_chanblock as f64,
+            freq: 150e6 + i_chanblock as f64,
         })
     }
     let chanblocks = Vec1::try_from_vec(chanblocks).unwrap();
@@ -490,7 +476,6 @@ fn calibrate_benchmarks(c: &mut Criterion) {
                     1e-4,
                     Polarisations::default(),
                     false,
-                    false,
                 );
             });
         }
@@ -562,9 +547,10 @@ fn io_benchmarks(c: &mut Criterion) {
     };
 
     // Open the readers.
+    let gpuboxes = [gpubox];
     let raw = RawDataReader::new(
-        &&metafits,
-        &[&gpubox],
+        &metafits,
+        &gpuboxes,
         None,
         RawDataCorrections::do_nothing(),
         None,
@@ -619,8 +605,8 @@ fn io_benchmarks(c: &mut Criterion) {
         |b| {
             // Need to re-make the reader with the new corrections.
             let raw = RawDataReader::new(
-                &&metafits,
-                &[&gpubox],
+                &metafits,
+                &gpuboxes,
                 None,
                 RawDataCorrections::default(),
                 None,

mdbook/src/SUMMARY.md

@@ -26,6 +26,7 @@
 - [Apply solutions](user/solutions_apply/intro.md)
   - [Simple usage](user/solutions_apply/simple.md)
 - [Plot solutions](user/plotting.md)
+- [Convert visibilities](user/vis_convert/intro.md)
 - [Simulate visibilities](user/vis_simulate/intro.md)
 - [Subtract visibilities](user/vis_subtract/intro.md)
 

mdbook/src/user/vis_convert/intro.md

@@ -0,0 +1,21 @@
+# Convert visibilities
+
+`vis-convert` reads in visibilities and writes them out, performing whatever
+transformations were requested on the way (e.g. ignore autos, average to a
+particular time resolution, flag some tiles, etc.).
+
+~~~admonish info title="Simple examples"
+```shell
+hyperdrive vis-convert \
+    -d *gpubox* *.metafits \
+    --tile-flags Tile011 Tile012 \
+    -o hyp_converted.uvfits hyp_converted.ms
+```
+
+```shell
+hyperdrive vis-convert \
+    -d *.uvfits \
+    --no-autos \
+    -o hyp_converted.ms
+```
+~~~