Skip to content

Overview of PyFluxPro

Jump to bottom
Peter Isaac edited this page Jun 2, 2022 · 7 revisions

Overview of PyFluxPro

General

PyFluxPro takes data recorded at a flux tower and process this data to a final, gap-filled product with Net Ecosystem Exchange (NEE) partitioned into Gross Primary Productivity (GPP) and Ecosystem Respiration (ER). The starting point is an Excel workbook containing flux tower data typically averaged over 30 or 60 minutes. The finish point is a collection of netCDF and Excel files containing gap filled and partitioned data at the tower time step and daily, monthly, annual and cumulative summaries. PyFluxPro can be used interactively, allowing users to fine tune the processing at each stage after reviewing plots of the intermediate data, or in a batch mode, where multiple processing stages can be linked together and performed without supervision.

Further descriptions of PyFluxPro are contained in 2 documents included in the PyFluxPro/docs directory: the Powerpoint presentation "An_Introduction_to_PyFluxPro.pptx" and a paper from the Biogeosciences OzFlux Special Issue in 2017 "bg-14-2903-2017.pdf".

Processing Levels

PyFluxPro is built around 6 processing stages:

  1. Level 1 (L1) - reads data from an Excel workbook, combines it with user-specified metadata and writes the data and metadata to an L1 netCDF file.
  2. Level 2 (L2) - reads the L1 netCDF file, applies the user specified quality control measures to the data, plots the pre- and post-quality controlled data (described in the Level 2 section) and writes the quality controlled data to an L2 netCDF file.
  3. Level 3 (L3) - reads the L2 netCDF file, applies the user specified post-processing to the quality controlled data (described in the Level 3 section), plots the post-processed data and writes it to an L3 netCDF file.
  4. Level 4 (L4) - reads the L3 netCDF file, gap fills the meteorological data using ancillary data from automated weather stations (AWS), numerical weather prediction (NWP) models and climatology, plots the gap-filled data and writes it to an L4 netCDF file.
  5. Level 5 (L5) - reads the L4 netCDF file, applies the u* filter to the selected flux data, gap fills the flux data using a neural network (SOLO) or Marginal Distribution Sampling (MDS), plots the gap-filled data and writes it to an L5 netCDF file.
  6. Level 6 (L6) - reads the L5 netCDF file, derives ER from the nocturnal, u*-filtered NEE data, trains 3 respiration models on the observed ER (SOLO, Lloyd-Taylor and Lasslop et al), partitions the gap-filled NEE into GPP and ER using the observed and modelled ER.

Levels 1 to 3 are typically done in chunks of a single year. Yearly L3 files can then be concatenated to a single file containing multiple years before applying the L4, L5 and L6 processing. However, PyFluxPro is flexible enough to allow users to process single years through all steps or to use multiple-year files for all stages. The user is free to process their data in whatever time chunks make sense to them.

Control Files

PyFluxPro uses text-based control files to specify which quality control and processing options are to be applied to which variables. The control files can be edited by the user within the application with a rich set of context-sensitive menus to guide the users choice of options.

The typical workflow for all processing levels is to open the appropriate control file in PyFluxPro, edit the control file using the right-click, context-sensitive menu or direct editing via the GUI to reflect the users processing choices, save any changes and then run PyFluxPro with the edited control file.

Detailed explanations of the control files and how to edit them are given in the sections for each processing level.

A Note About Time

The whole area of data logger time stamps and time zones is complicated and can get messy very quickly. This section explains a few things about the way PyFluxPro uses time:

  1. PyFluxPro never alters the relationship between your data and its timestamp. That is sacrosanct! The timestamp that your data has when it is read in at L1 is always preserved.
  2. The time zone of your site is entered in the Global section of the L1 control file and becomes a global attribute in the L1 and subsequent netCDF files. PyFluxPro does not use the time zone but it does check the specified time zone against the site latitude and longitude to make sure they are consistent. A full list of time zones in the format expected by PyFluxPro is available at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones.
  3. L1 to L3, L5 and L6 processing make no assumptions about the timestamp of your data. When using alternate data to gap fill meteorological data at L4, PyFluxPro assumes that the timestamp of the flux tower data and the alternate data are in the same time zone. TERN Ecosystem Processes Central Node supplies alternate data from automatic weather stations, numerical weather prediction models and reanalyses for each TERN/OzFlux site in local time for that site.
  4. PyFluxPro makes no assumption about the timestamp referring to the start or end of the averaging period. However, the alternate data supplied by TERN EPCN for gap filling at L4 uses the convention that the timestamp refers to the end of the averaging period. To be consistent with this, the timestamp of the data at L1 should also refer to the end of the averaging period. NOTE: the gap filling at L4 removes any phase difference between the alternate and flux tower data by doing a lagged correlation before the least squares fit and this should deal with any timestamp inconsistencies but it is best to not rely on this feature.
  5. Date and times used internally by PyFluxPro do not have a time zone associated with them. But we think we should make PyFluxPro timezone aware as soon as possible. What do you think?
  1. Home
  2. FAQ
  3. Installing PyFluxPro
    1. Installing from GitHub
    2. Running PyFluxPro
    3. Updating PyFluxPro
    4. Removing PyFluxPro
  4. Using the Examples
  5. Overview of PyFluxPro
    1. General
    2. Processing Levels
    3. Control Files
    4. Typical Workflow using PyFluxPro
    5. A Note About Time
  6. The PyFluxPro GUI
  7. Editing control files
  8. Variable names and attributes
    1. Introduction
    2. Variable naming: rules
    3. Variable attributes: units, long_name and standard_name
    4. Variable attributes: instrument
  9. Quality control flag definitions
  10. Level 1
    1. Overview
    2. The L1 Control File
      1. Files Section
      2. Global Section
        1. A note about time_zone
      3. Variables Section
    3. Running L1
    4. Output from Running L1
  11. Level 2
    1. Overview
    2. The Quality Control Checks
    3. The L2 Control File
      1. The Files Section
      2. The Options Section
      3. The Variables Section
      4. The Plots Section
    4. Running L2
    5. Output from Running L2
  12. Level 3
    1. Overview
    2. A Little History
    3. L3 Processing Steps
    4. The L3 Control File
      1. The Files Section
      2. The Options Section
      3. The Soil Section
      4. The Massman Section
      5. The Variables Section
      6. The Plots Section
    5. Running L3
    6. Output from Running L3
  13. Level 4
    1. Overview
    2. Background
    3. The L4 Control File
      1. Files Section
      2. Options Section
      3. Drivers Section
    4. Running L4
    5. Output from Running L4
  14. Level 5
    1. Overview
    2. Background
    3. The L5 Control File
      1. Files Section
      2. Imports Section
      3. Options Section
      4. u* Threshold Section
      5. Fluxes Section
    4. Running L5
    5. Output from Running L5
  15. Level 6

Clone this wiki locally