Code Modules

Code Modules

The Code/ directory contains processing scripts, QA tools, and one-time utilities. Scripts are organised into prefixed subdirectories.

Directory Structure

Code/
├── DS02_DatasetQA/          # Dataset quality assurance scripts
│   ├── QA00_SpectralValidation.py
│   ├── QA02_SpectralRunComparison.py
│   └── README.md
├── DS03_PlotExtractionCode/  # Plot-level data extraction
│   └── PE00_LIDAR_extraction.py
├── functions/                # Shared utility functions
│   ├── core_functions/
│   │   ├── parse_APPN_dataset_path.py
│   │   ├── outputs_up_to_date.py
│   │   └── tests/
│   │       └── test_parse_APPN_dataset_path.py
│   └── spectral_qc/          # Shared spectral-QC helpers (QA00/QA02)
└── OT00_OneTimeScripts/      # One-time migration/maintenance scripts
    ├── OT00_RenameTiertoT.py
    └── OT01_MoveGrawToRaw.py

Naming Convention

Scripts use a prefix system:

Prefix	Meaning	Example
DS	Dataset processing step	DS02_DatasetQA
QA	Quality assurance	QA00_SpectralValidation.py
PE	Plot extraction	PE00_LIDAR_extraction.py
OT	One-time script	OT00_RenameTiertoT.py, OT01_MoveGrawToRaw.py

---

QA00 — Spectral Validation (per-run extraction + QC report)

File: Code/DS02_DatasetQA/QA00_SpectralValidation.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v2.2 (13.08.2026)

Extracts spectral data from ELM/VAL reflectance panels in hyperspectral imaging datasets and produces a per-run QC report and figure. Panel files must follow the official AerialDataQC naming convention. The summary below covers the essentials — see the linked README for tested package versions, full folder-structure assumptions, output schemas, troubleshooting, and future enhancements.

What It Does

1. Searches for QC panel vector files matching QC_{ELM|VAL}[_{id}]_Panels[_{extra}].geojson (or .shp) under T1_proc/QC_data/
2. Locates corresponding VNIR and SWIR orthomosaic rasters
3. Extracts per-pixel reflectance with per-band wavelengths (nm) into QC_data/QC_Spectral_Tables/QC_{ELM|VAL}[_{id}]_spectra_{VNIR|SWIR}[_gproN].{parquet|csv}
4. Identifies the physical panel set from its Panel_ref signature (panel_set column: Gryfn4P/Gryfn2P/unknown)
5. Writes a per-run QC report (QC_data/QC_spectra_report.json) and per-target spectra figure (QC_data/QC_plots/)
6. Skips up-to-date outputs (mtime caching); prints a REPORTED/SKIPPED summary table

Command-Line Arguments

Argument	Description
--path	Path to search for QC panel files (default: git root)
-f, --force	Force overwrite of existing output files
--type	Output format: parquet (default) or csv
-s, --skipplot	Skip per-run figure generation
--skip-processing	Never process rasters; only load existing outputs for reporting
--exclude-dir	Directory names to exclude from the panel search
--no-radiance-check	Disable the reflectance vs radiance range check
--keep-xy	Retain per-pixel x/y coordinate columns
--allow-multi-gpro	Process runs with multiple .gpro folders (debugging only)
-v, --verbose	Detailed output

Usage

# Standard run (extraction + report + figure per run)
python Code/DS02_DatasetQA/QA00_SpectralValidation.py --path /path/to/data

# Extraction only, skip figures
python Code/DS02_DatasetQA/QA00_SpectralValidation.py --path /path/to/data -s

---

QA02 — Spectral Run Comparison (multi-run figures + sharing)

File: Code/DS02_DatasetQA/QA02_SpectralRunComparison.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v1.1 (13.08.2026)

Gathers the extracted spectra tables produced by QA00 across every run under a path and produces cross-run comparison figures (per-panel reflectance and residual spectra, one line per run). Never opens raster files. Residuals are each run's deviation from the cross-run mean spectrum at each snapped wavelength.

Command-Line Arguments

Argument	Description
--path	Node or project folder to crawl (default: git root)
--output-dir	Explicit figure directory (required for other path levels)
--no-save	Display figures instead of saving
--type	Table format: parquet (default) or csv
--load-dir	Also load spectra tables from this folder (other nodes)
--save-dir	Save copies of every gathered table for sharing
--start-date / --end-date	Inclusive date window limiting which runs are compared
--errorbar	Spread band per run line: pi (default), sd, none
--exclude-dir	Directory names to exclude from the table search
-v, --verbose	Detailed output

Usage

# Compare every run under a node (figures -> <Node>/Documents/QCReports/)
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node

# Share with other nodes / combine external data
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node --save-dir /shared/spectra
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node --load-dir /shared/spectra

# Only compare runs after 1 June 2026
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node --start-date 2026-06-01

Dependencies

numpy, pandas, xarray, rioxarray, rasterio, geopandas, shapely, matplotlib, seaborn, tqdm, GitPython, palettable (+ colorcet for >20 runs, pyarrow for parquet)

See the DS02_DatasetQA README for tested package versions and conda environment setup.

---

PE00 — LIDAR Plot Extraction

File: Code/DS03_PlotExtractionCode/PE00_LIDAR_extraction.py
Full documentation: Code/DS03_PlotExtractionCode/README.md
Authors: Arden Burrell & Richard Harwood
Version: v1.0 (09.03.2026)

Extracts plot-level data from LIDAR point clouds (GOBI and CALVIS sensors) by clipping to plot shapefiles and attaching DSM/DTM elevations. The summary below covers the essentials — see the linked README for full folder-structure assumptions, output schemas, troubleshooting, and future enhancements.

What It Does

1. Discovers FieldLog.csv files (or crawls the hierarchy) to build a table of GOBI/CALVIS acquisitions
2. Loads and validates plot-layout shapefiles under Documentation/Plot_Layout/
3. Locates matching *LiDAR_CombinedPointCloud.las, *LiDAR_DSM_*.tif, *LiDAR_DTM_*.tif files
4. Spatial-joins the point cloud to plot polygons and extracts DSM/DTM values per point
5. Computes canopy height (Delta_z = z - DTM)
6. Writes CSV/Parquet tables plus a YAML metadata sidecar (runtime, system, git state)
7. Optionally saves identifiable copies to a shared --save-dir

Command-Line Arguments

Argument	Description
--path	Root path to search (default: git root)
--path-level	Level that --path refers to: root, node, project, site, sensor, date, run (default: site)
-f, --force	Force overwrite of existing output files
--savetype	Output format: csv or parquet (default: parquet)
--save-dir	Save identifiable copies of extracted data for sharing
--load-dir	Load previously extracted data from another node
-v, --verbose	Detailed output

Usage

# Standard run
python PE00_LIDAR_extraction.py

# Process a single site
python PE00_LIDAR_extraction.py --path /data/node/project/2026SiteA --path-level site

# Save shared copies for other nodes
python PE00_LIDAR_extraction.py --save-dir /shared/lidar

Dependencies

numpy, pandas, xarray, rioxarray, laspy, geopandas, shapely, pyyaml, tqdm, GitPython

---

OT00 — Rename Tier to T

File: Code/OT00_OneTimeScripts/OT00_RenameTiertoT.py

A one-time migration script that renames folders starting with Tier to start with T (e.g. Tier0_raw → T0_raw).

Usage

# Use git root as search path
python OT00_RenameTiertoT.py

# Specify a custom path
python OT00_RenameTiertoT.py --path /path/to/data

The script searches recursively, lists all matching folders, asks for confirmation, then performs the rename.

---

OT01 — Move .graw Folders from T1_proc to T0_raw

File: Code/OT00_OneTimeScripts/OT01_MoveGrawToRaw.py

A one-time migration script that relocates *.graw folders from T1_proc into the adjacent T0_raw folder for GOBI and CALVIS sensors only.

What It Does

1. Walks the search root looking for directories named T1_proc
2. Keeps only those whose ancestry includes a folder whose name contains GOBI or CALVIS (case-insensitive)
3. Collects the direct child folders of T1_proc ending in .graw
4. Lists every proposed move and asks for confirmation
5. Moves each .graw folder into the sibling T0_raw/ (created if missing), refusing to overwrite existing targets

Layout Assumption

.../<sensor>/<date>/run_XX/T1_proc/<name>.graw
        ->
.../<sensor>/<date>/run_XX/T0_raw/<name>.graw

Command-Line Arguments

Argument	Description
--path	Optional root directory to search. When supplied, the git repository check is skipped.
--dry-run	List candidate moves without performing them.

Usage

# Use git root as search path
python OT01_MoveGrawToRaw.py

# Preview moves only
python OT01_MoveGrawToRaw.py --dry-run

# Specify a custom path
python OT01_MoveGrawToRaw.py --path /path/to/data

Dependencies

tqdm (and git on PATH when --path is not provided).