Level 2
Level 2, or L2 for short, is the stage where PyFluxPro allows you to remove what you believe to be bad data and to plot the results of the quality control process so you can make sure you are doing something sensible. L2 reads an L1 netCDF file, produces an L2 netCDF and plots of the variables before and after quality control.
Eddy covariance is an inherently noisy measurment technique and the "noisiness" or signal to noise ratio varies a great deal from site to site and even from season to season within a single site. In general, it's not possible to use flux tower data without some sort of quality control and this is especially true when you get to the later stages of processing, gap filling and partitioning NEE into GPP and ER. More complex processing steps often act as "noise amplifiers".
Programs that calculate the average fluxes from 10 or 20 Hz data, such as EddyPro, often have very good schemes for detecting bad data based on various statistics compiled from the 10 or 20 Hz data and these can be a good indicator of data quality. Often, though, data which is obviously bad is still present even when these objective methods are applied. Also, we often have to clean up the non-flux data as well. Sensors fail in many ways and sometimes they continue to produce data that looks similar to good data but is, in fact, bad data. We need some general approaches to cleaning up the data from flux towers and PyFluxPro provides these approaches.
A word of caution. Many of these quality control techniques are subjective, not objective. They allow you to remove what you consider to be bad data so that later processing stages, particularly those based on statistical techniques such as u* threshold detection and gap filling, will work effectively. However, their subjective nature makes them dangerous. It is possible to produce very clean-looking, smooth data sets using these quality control techniques but in the process you may remove a lot of good data simply because it doesn't conform to your idea of how your ecosystem shoud behave. Use these techniques carefully, not blindly, and always ask yourself if your processing methods support your intended analysis.
The general quality control method is as follows:
- The user specifies the quality control checks they want to apply in the L2 control file. The quality control checks must be applied to each individual variable.
- When L2 is run, PyFluxPro will check each data value against the quality control checks you have specified in the L2 control file;
- If the data value passes the quality control check, it is left as is and the quality control flag for that data value (see Overview of PyFluxPro) is left as 0 (good data).
- If the data value fails the quality control check, the data value is set to -9999 and the quality control flag for that data value is set to the unique value for that quality control check.
The quality control checks available in PyFluxPro are described below:
- DependencyCheck rejects data values for this variable depending on the state of one or many prerquisite variables. If any of the prerequisite variables fail their quality control checks then the data value for this variable is also rejected.
- DiurnalCheck rejects data values that are more than a specified number of standard deviations away from the mean value for this hour of the day. The user specifies the number of standard deviations. If you enter a single value for the number of standard deviations, the single value is used for the whole data set. You can enter 12 values separated by commas to specify different values for each month of the year.
- ExcludeDates allows the user to reject data based on one or more date ranges.
- ExcludeHours allows the user to reject data for specified hours of the day for one or more date ranges.
- LowerCheck rejects data that lies below a lower bound for an arbitrary number of date ranges, the lower bound is piece-wise linear.
- RangeCheck rejects data values that lie outside the specified lower and upper ranges specified by the user. If you enter a single value for the lower and upper limits, the single value is used for the whole data set. You can enter 12 values separated by commas to specify different values for each month of the year.
- UpperCheck rejects data that lies above an upper bound for an arbitrary number of date ranges, the upper bound is piece-wise linear.
In addition to the quality control checks, PyFluxPro can apply 2 functions at L2:
- CorrectWindDirection allows the user to add an offset to wind directions for an arbitrary number of date ranges.
- Linear allows the user to apply a linear correction to any variable for an arbitrary number of data ranges.
PyFluxPro can also automatically apply quality control checks based on the sonic anemometer data (SONIC_Check) and the IRGA data (IRGA_Check). Both of these are types of dependency checks, see below:
- SONIC_Check uses the sonic anememeter diagnostic word, standard deviations of vertical wind speed and virtual temperature. If either of these quantities are outside the ranges specified by the user then all variables that use data from the anemometer are rejected. These include the covariances between the wind speed components (Ux, Uy and Uz) and the scalars (AH, CO2 and Ta).
- IRGA_Check uses the IRGA diagnostice work, the H2O and CO2 signal strength or the AGC and the standard deviations of H2O and CO2 concentrations. If any of these quantities are outside the ranges specified by the user then all variables that use data from the IRGA are rejected. These include the covariances between the wind speed components (Ux, Uy and Uz) and the scalars (AH, CO2 and Ta).
The SONIC_Check and IRGA_Check can be disabled, see the description of the Options section below.
The L5 control file consists of the following sections:
- Files
- Options
- Variables
- Plots
The contents of these sections and how to edit them are described below.
The Files section allows the user to specify the path to the input and output files, the names of the input and output files and the path for plots generated by the L2 processing, see the screenshot below.
The entries in the Files section are as follows:
- file_path - the path to the data files
- in_filename - the input file name
- out_filename - the output file name
- plot_path - the path for plots generated by the L2 processing
The entries in Files section can be edited by right clicking on the entry in the Value column and using the Browse... feature or by double clicking on the entry in the Value column and manually entering the required text.
The Options section allows the user to specify the options that control some aspects of the L2 processing.
The Options section contains an entry called irga_type with the default value of Li-7500. You can right click on the irga_type value to display a list of options for irga_type, see below.
A list of the options available can be displayed by right clicking on the Options section title in the Parameter column, see the screenshot below.
The SONIC_Check and IRGA_Check options allow the user to turn off the default SONIC and IRGA checks.
The Variables section is where the user specifies the quality control checks to be applied to the variables read in at L1. An example of a variable subsection from the Loxton control file is shown below.
The name of the variable sub-section in the Parameter column, AH_HMP_10m in this case, is the variable name in the netCDF file.
Each quality control check forms a sub-section within the variable section and each of the quality control checks has additional entries that specify how the check will be used. In the example above, the RangeCheck will be applied with a lower limit of 0.0 and an upper limit of 35.0. You can specify a different limit for each month of the year by entering 12 values separated by commas.
You can add quality control checks to a variable by right clicking on the variable name to display a context menu with the available quality control checks. Select one and it will be added to the variable section.
The sections below desribes the options for each of the quality control checks.
The example above makes the AH_HMP_10M variable dependent on the Ta_HMP_10m variable. Any records rejected from Ta_HMP_10m will laso be rejected from AH_HMP_10m.
The example above will reject any AH_HMP_10m data that lies outside of the range +/- 5 standard deviations about the mean for the the time of day. For example, the mean and standard deviation are calculated for all data at 06:00 and any points at 06:00 that are outside +/- 5 standard deviations of that mean are rejected.
The example above will exclude all data between 2021-01-01 00:30 and 2021-02-01 00:00 i.e. the month of January. You can add additional date ranges by right clicking on ExcludeDates and selecting Add date range.
The example above will exclude all data for the hours of 12:30 and 13:00 between 2021-01-01 00:30 and 2021-02-01 00:00 i.e. the month of January. You can add additional date ranges by right clicking on ExcludeHours and selecting Add hour range.
The example above will apply:
- A lower limit to the data that starts at 0.0 on 2021-01-01 and rises to 10.0 on 2021-02-01.
- An upper limit to the data that starts at 30.0 on 2021-01-01 and drops to 20.0 on 2021-02-01.
You can add additional date ranges by right clicking on LowerCheck or UpperCheck and selecting Add date range.