Skip to content

Key Files

Jump to bottom
Arden Burrell edited this page May 5, 2026 · 3 revisions

Key Files

Reference for every metadata, configuration, and log file involved in the APPN Generic File Storage workflow. This includes the configuration files read by ProjectBuilder and every CSV, YAML, JSON, and TXT file that the script creates or maintains on disk.

File Summary

File Type Location Created By Purpose
NodeSummary.yaml YAML Repository root User (template provided) Defines nodes and their sensor platforms
FolderStructureInfo.txt TXT Repository root Manual reference Human-readable specification of the folder naming convention
{NodeName}_ProjectsSummary.csv CSV ./{NodeName}/ ProjectBuilder.py Boolean matrix of projects vs. sensor platforms
ProjectSummary.yaml YAML ./{NodeName}/{Project}/ ProjectBuilder.py Project metadata, researcher, and site definitions
FieldLog.csv CSV ./{NodeName}/{Project}/ ProjectBuilder.py Log of every field data collection event for a project
FieldNotes.txt TXT .../{Sensor}/{YYYYMMDD}/ ProjectBuilder.py Free-form field notes for a single collection day
RunOverview.csv CSV .../{Sensor}/{YYYYMMDD}/ ProjectBuilder.py Per-run status table for a single collection day

Note

No JSON files are created by the current scripts. This page will be updated if any are added in the future.

Important

The following key files are not yet documented on this page and still need to be added:

  • Plot delineation shapefiles (Documentation/Plot_Layout/*.shp)
  • QC data files (T1_proc/QC_data/, including QC panel shapefiles *QC_*_Panel*.shp)

NodeSummary.yaml

Type: YAML  |  Location: repository root  |  Maintained by: user

The primary configuration file. Defines the APPN nodes and the sensor platforms available at each. Read by ProjectBuilder.py at startup. See NodeSummary-Configuration for the full schema.

nodes:
  - name: "USYD_Narrabri"
    university: "University of Sydney"
    location: "Narrabri, NSW, Australia"
    SensorPlatforms:
      - GOBI
      - HIRES
      - M3M

FolderStructureInfo.txt

Type: TXT  |  Location: repository root  |  Maintained by: user

A plain-text reference document that describes the folder naming convention (node, project, site, sensor, date, run, tier). Not read by any script — kept as on-disk documentation for collaborators. See Folder-Structure for the rendered version.

{NodeName}_ProjectsSummary.csv

Type: CSV  |  Location: ./{NodeName}/  |  Created by: NodeChecker() in ProjectBuilder.py

A boolean matrix mapping projects (rows) to sensor platforms (columns). The user fills this in to declare which sensors are valid for which project. New columns are added automatically when sensors are added to NodeSummary.yaml.

Project GOBI HIRES M3M CALVIS
2025_Chickpea_E_Salter TRUE FALSE FALSE TRUE
2025_SIFcalibration_I FALSE TRUE TRUE FALSE

Note

  • The first column (index) is the project folder name.
  • Values must be TRUE or FALSE.
  • Missing sensor columns are added automatically (filled with False) when NodeSummary.yaml changes.
  • Project folders that exist on disk but are missing from this CSV trigger an orphan warning (see ProjectBuilder § Orphan Project Detection).
  • Re-running with --enable-sensors will flip a FALSE to TRUE automatically when a FieldLog.csv row references that sensor.

ProjectSummary.yaml

Type: YAML  |  Location: ./{NodeName}/{Project}/  |  Created by: _projYAML() in ProjectBuilder.py

Per-project metadata: researcher, funding, status, and the list of sites belonging to the project. Created from the template in _defaultProjectYAML(). Missing keys are added on subsequent runs without overwriting existing values.

project:
  ShortName: "2025_Chickpea"
  FullName: ""
  description: ""
  start_date: ""
  end_date: ""
  funding_source: ""
  status: ""
  ProjectCode: ""
  Internal: null
  researcher:
    FirstName: ""
    LastName: ""
    Title: ""
    email: ""
    institution: ""
    role: "Principal Investigator"
    orcid: ""
  sites:
    - name: ""
      year: -9999
      season: ""
      SubLocation: ""
      latitude: .nan
      longitude: .nan
      description: ""
      ControlledEnvironment: null
      sensors: []

Key fields

Field Type Notes
ShortName str Mirrors the project folder name
Internal bool / null True/False matches the _I / _E folder suffix
researcher.* str Lead researcher contact details
sites[].name str Site identifier — must match Site values in FieldLog.csv
sites[].year int Must match Year in any FieldLog.csv row referencing the site
sites[].ControlledEnvironment bool / null True → site folder gets _C, False → _F, null → no suffix
sites[].sensors list Sensors deployed at the site (e.g. [GOBI, HIRES])

FieldLog.csv

Type: CSV  |  Location: ./{NodeName}/{Project}/  |  Created by: projBuilder() in ProjectBuilder.py

One row per field data collection event. Drives folder creation for sensor / date / run / tier sub-trees. Validated by Rowchecker() on every run.

Column Type Description
Year int Year of collection
Month int Month of collection
Day int Day of collection
Sensor str Sensor platform name (must be TRUE in {NodeName}_ProjectsSummary.csv)
Technician str Name of the technician who collected the data
Runs int Number of runs (≥ 1); creates run_00 … run_{N-1} folders
Site str Site name; must match a site in ProjectSummary.yaml with the same year
MakeNotesFile bool Create FieldNotes.txt for this day. Defaults to True when blank/NaN
MakeTableFile bool Create RunOverview.csv for this day. Defaults to True when blank/NaN
CheckSum float Auto-computed integrity hash for the row

Note

  • The MakeNotesFile and MakeTableFile columns accept false, f, 0, no, n (case-insensitive) to skip creation. Anything else (including blanks) resolves to True. The resolved boolean is written back into the CSV and the checksum is recomputed.
  • Dates older than 14 days require the --historical flag.
  • If column schema changes (e.g. new columns added), existing checksums are cleared and recomputed automatically.

FieldNotes.txt

Type: TXT  |  Location: ./{NodeName}/{Project}/{Site}/{Sensor}/{YYYYMMDD}/  |  Created by: Sitebuilder() in ProjectBuilder.py

An empty plain-text file created alongside each run_XX/ set, intended for free-form notes recorded during the field day (weather, issues, equipment changes, etc.). Created with pathlib.Path.touch(), so it will not overwrite an existing file. Suppressed when MakeNotesFile is False for the corresponding FieldLog.csv row.

RunOverview.csv

Type: CSV  |  Location: ./{NodeName}/{Project}/{Site}/{Sensor}/{YYYYMMDD}/  |  Created by: Sitebuilder() in ProjectBuilder.py

A per-run status table created alongside the run_XX/ folders. One row per run with at minimum a RunFailed boolean column for marking failed acquisitions.

Run RunFailed
run_00 False
run_01 False

Note

  • The index column (Run) holds the run folder names.
  • Existing RunOverview.csv files are never overwritten — users may add extra columns and they will be preserved.
  • Suppressed when MakeTableFile is False for the corresponding FieldLog.csv row.

Where Each File Lives

repo_root/
├── NodeSummary.yaml                                      # config (user)
├── FolderStructureInfo.txt                               # reference (user)
└── {NodeName}/
    ├── {NodeName}_ProjectsSummary.csv                    # node-level
    └── {Project}/
        ├── ProjectSummary.yaml                           # project metadata
        ├── FieldLog.csv                                  # project-level log
        └── {Site}/{Sensor}/{YYYYMMDD}/
            ├── FieldNotes.txt                            # day-level notes
            ├── RunOverview.csv                           # day-level run table
            └── run_XX/{T0_raw,T1_proc,T2_traits}/        # data tiers

Related Pages

Repository · Issues · MIT License · See Contributing-to-the-Wiki to edit these pages.

APPN DataStorage Wiki

Start here

APPN Folder Structure

Guides

Reference

Project

Clone this wiki locally