CLI_var_min_max

Minimum/Maximum checker

Checks whether values in user-defined variable(s) in a NetCDF fits within a certain range of acceptable min/max values specified within a YAML configuration file.

Table of Contents

• Syntax
• Arguments
• Configuration File Structure
• Example

---

Syntax

nes check_min_max -i <input_file.nc> -c <config_file.yaml> [--avoid_first_hours N]

Arguments

• -i, --input_path: Path to the NetCDF file to validate. (required)
• -c, --config_path: Path to the YAML configuration file. (required)
• --avoid_first_hours N (int): Skips the first N hours.

Configuration File Structure

variable1:
    min_abs:
    max_abs:

    min_lower:
    min_upper:

    max_lower:
    max_upper:

variable2:
    min_abs:
    max_abs:

    min_lower:
    min_upper:

    max_lower:
    max_upper:

• variable: Name of the variable in the NetCDF to perform the check on, must match exactly with how it's specified within the NetCDF.
• min_abs: Absolute minimum cut-off value to be considered an error. If a value is found below this number the program will finish with an error. Usually set to 0 to detect either NaNs or negative values.
• max_abs: Absolute maximum value to be detected in the NetCDF, any values above will be considered erroneous. If a value is found above this number the program will finish with an error. Usually set to a very high number to detect infinite values.
• min_lower: Lower bound of acceptable minimum value range. If the minimum value of the NetCDF is below this value, the program returns a warning.
• min_upper: Upper bound of acceptable minimum value range. If the minimum value of the NetCDF is above this value, the program returns a warning.
• max_lower: Lower bound of acceptable maximum value range. If the maximum value of the NetCDF is below this value, the program returns a warning.
• max_upper: Upper bound of acceptable maximum value range. If the maximum value of the NetCDF is above this value, the program returns a warning.

Essentially there are two primary checks happening here:

• The program first checks for the absolute smallest and largest values that are physically possible (defined by min_abs and max_abs), if the minimum and maximum of the NetCDF are smaller or larger, respectively, the program will return an error because we are "certain" that there is something wrong in the values of the dataset. No error is returned if: min_{nc} > min_{abs} \quad \text{and} \quad max_{nc} < max_{abs} Where min_{nc} and max_{nc} are the minima and maxima of the NetCDF, respectively, and min_{abs} and max_{abs} are the absolute minima and maxima provided in the YAML configuration file.

• The secondary check is to create a range of minimum and maximum values, inside of which a value is considered "correct", and an outlier if outside, for which the program will throw a warning:

  

  Ideally, the range of minima and maxima should be constructed by analyzing a large amount of data for a variable in several files ahead of time, using the lowest and highest values of both minima and maxima encountered in this dataset to construct a range of acceptable values. The program will not return a warning if the conditions:

  min_{ lower } \leq min_{ nc } \leq min_{ upper } \quad \text{and} \quad max_{ lower } \leq max_{ nc } \leq max_{ upper } are met. Where min_{lower}, min_{upper}, max_{lower} and max_{upper} are the upper and lower bounds of the minima and maxima provided in the YAML configuration file.

Example

Check variable sconcco with provided YAML file and skip first two timesteps

nes check_min_max -i input.nc -c config.yaml --avoid_first_hours 2

config.yaml

sconcco:
  min_abs: 0.0
  max_abs: 10.0

  min_lower: 0.01
  min_upper: 0.07

  max_lower: 2.706
  max_upper: 5.0