SCOS Actions 3.0.0

[aromanielloNTIA]

The major components of this update are:

• Increments minimum Python version from 3.7 to 3.8
• Refactors code such that signal processing, metadata, and action files are clearly organized
• Adds a variety of reusable signal processing routines optimized for large-array calculations
• Refactors existing actions to leverage new signal processing routines, increasing legibility, consistency, and performance
• Changes Y-factor action from FFT-based to time domain-based, and adds optional IIR filtering before Y-factor
• Above changes are implemented without affecting behavior of existing actions
• Improves some code documentation, especially in the signal processing modules
• Updates development tooling which seems to have been unused for some time, in the hopes that future updates will leverage the included pre-commit tools again
• Updates NumPy and Django minimum versions to patch vulnerabilities GHSA-fpfv-jqm9-f5jm, GHSA-8x94-hmjh-97hq, and GHSA-p64x-8rxx-wf6q (addressing Bump numpy from 1.21.5 to 1.22.0 #21 and Bump django from 3.2.12 to 3.2.15 #31)
• Version number bumped to 3.0.0 since these changes are technically not backwards compatible, since the package structure has substantially changed (i.e. import scos_actions.actions.fft would need to be changed to import scos_actions.signal_processing.fft in any software depending on scos_actions)
• Switch the project's backend to Hatchling

Testing

All tests are passing and the changes have been tested using a mock USRP and the seadog03 sensor (Tektronix 507A). Running scos-sensor branch actions-dsp-refactor will point a USRP sensor to this branch (and use scos-usrp branch actions-dsp-update. The actions-dsp-refactor-tek does the same for a Tektronix sensor (using scos-tekrsa branch actions-dsp-update).

A much more descriptive list of changes:

• Dependency/development tooling updates:
  • Updates minimum Python version to 3.8 (from 3.7)
  • Adds new dependency: NumExpr. Set minimum version 2.8.3 (which is current)
  • Increase minimum SciPy version to 1.8.0 (for parallelized FFT computation with scipy.fft.fft added in 1.6.0, and to support NumPy higher than 1.22.0)
  • Increase minimum NumPy version to 1.22 (fixes GHSA-fpfv-jqm9-f5jm, addresses Bump numpy from 1.21.5 to 1.22.0 #21)
  • Increase minimum Django version to 3.2.15 (fixes GHSA-p64x-8rxx-wf6q and GHSA-8x94-hmjh-97hq, addressing Bump django from 3.2.12 to 3.2.15 #31)
  • Removes development dependency black, since pre-commit specifies the version and installs it as needed.
  • Adds development dependency pip-tools, since pip-tools is recommended for dependency management in the README
  • Updated .pre-commit-config.yaml: Fixed broken Markdown linting, updated black version, switched isort source to non-deprecated repository, removed deprecated seed-isort-config. Added pyupgrade hook and multiple other default hooks.
• Major changes/refactoring to signal processing routines used by actions
  • Added signal_processing/apd.py with routines for estimating the APD with parameterized downsampling
  • Added signal_processing/power_analysis.py with routines for time domain power calculations, generalized power detectors (frequency or time domain), and quantile filtering
  • Added signal_processing/unit_conversion.py with routines for unit conversion, which now use NumExpr to speed up computation on large arrays
  • Added signal_processing/filtering.py with routines for digital filtering. Currently contains routines to generate IIR and FIR filters using SciPy, as well as a few useful filter analysis methods.
  • Moved get_enr and get_temperature out of the Y-factor action, and into signal_processing/calibration.py, alongside the Y-factor calculation
  • Moved /scos_actions/actions/fft.py to /scos_actions/signal_processing/fft.py
  • Updated fft.py to contain a generally-applicable set of reusable FFT-related routines:
    • Separated functionality into dedicated methods which can be individually used when different types of FFT results are desired
    • Removed all power scaling, as those decisions are application-dependent and should be contained in the action definition. This creates more legible actions and allows for a more generalized set of FFT routines.
    • fft.get_fft_frequencies now returns a list instead of a NumPy array, ready to be stored in metadata without conversion in each action.
    • fft.get_fft_window now supports many more window types using scipy.signal.get_window(), and is also backwards-compatible for previously allowed inputs like 'Flat Top' or 'Hanning' ('flattop' and 'hann', respectively, in SciPy syntax)
    • Switched from numpy.fft.fft to scipy.fft.fft to leverage parallel processing for faster FFT computation
    • Moved non-FFT-related code into unit_conversion.py or power_analysis.py as applicable
    • Updated and added documentation
  • Generalized detectors in power_analysis.py:
    • Generalized detectors can apply any combination of min/max/mean/median/sample detectors to either frequency- or time-domain data
    • Replaced single instance of FFT M4S detector enumeration with a general way to create and apply detector enumerations. This supports future actions more easily by being able to define any combination of the M4S detectors as a new detector. Create a detector using power_analysis.create_power_detector(). Apply to power samples using power_analysis.apply_power_detector().
• Updated existing actions for new signal processing library
  • Results in more legible action definitions, with power scaling clearly specified in the action to make sure the result is as expected and that no unnecessary computation is performed (e.g. applying power scaling in fft.py which then has to be reversed in the action definition).
  • Actions now make heavier use of instance variables, to reduce redundant computation (e.g., the FFT window is no longer generated repeatedly every time the action is run with the same parameters).
• Consolidated and improved utils:
  • Removed confusing actions/action_utils.py and moved its functionality into utils.py
  • Replaced old methods from action_utils.py with a more general get_parameter() method, which includes basic error handling.
  • Replaced get_iteration_parameters with get_iterable_parameters, which now allows for creating an iterable result for single-frequency measurements as well as replicating values which are specified once for multi-frequency measurements (i.e. a multi-frequency action can now include one gain value and multiple frequencies, and the gain value will be used for all iterations). This also results in simplified multi-frequency action definitions.
• Fixed use of deprecated Django arguments in actions/interfaces/signals.py which caused warnings when running tests

[dboulware]

Looks really good. See minor comments in line.

[dboulware]

Looks good!