ProjectBuilder

ProjectBuilder

ProjectBuilder.py is the main automation script that generates and manages the APPN folder structure. It reads configuration from YAML and CSV files, creates folders, field logs, and project metadata files, and optionally commits changes to git.

Author: Arden Burrell
Version: v1.0 (22.05.2025)

Usage

python ProjectBuilder.py [options]

Command-Line Arguments

Argument	Type	Default	Description
--no-git	flag	False	Disable all git operations (pull, add, commit, push)
--projectsYAML	str	./NodeSummary.yaml	Path to the node YAML file with sensor definitions
-p, --historical	flag	False	Allow historical dates (older than 14 days) in field logs

Examples

# Standard run with git integration
python ProjectBuilder.py

# Run without git (useful for testing or offline work)
python ProjectBuilder.py --no-git

# Allow adding field log entries with dates older than 14 days
python ProjectBuilder.py --historical

# Use a custom node YAML
python ProjectBuilder.py --projectsYAML ./custom_nodes.yaml

How It Works

The script follows this sequence:

1. Git pull — Syncs the local repo with the remote (unless --no-git)
2. Load node configuration — Reads NodeSummary.yaml for node and sensor definitions
3. For each node:
   • Creates the node folder if missing
   • Creates or loads {NodeName}_ProjectsSummary.csv
   • Ensures all sensor columns exist in the CSV
4. For each project in the CSV:
   • Creates Documentation/ and Code/ subfolders
   • Creates or loads ProjectSummary.yaml with project metadata
   • Creates or loads FieldLog.csv for tracking field days
   • Creates site folders based on the project YAML
5. For each field log entry:
   • Validates the row (date, sensor, site, run count, checksum)
   • Creates the sensor → date → run → tier folder structure
   • Optionally creates FieldNotes.txt
   • Updates checksums in the field log
6. Git commit and push — Commits all new files and pushes (unless --no-git)

Generated Files

{NodeName}_ProjectsSummary.csv

A boolean matrix mapping projects to sensor platforms. Each row is a project, each column is a sensor. Values should be TRUE or FALSE.

Project	GOBI	HIRES	M3M	CALVIS	...
2025_Chickpea	TRUE	FALSE	FALSE	TRUE	...

ProjectSummary.yaml

Created in each project folder with template fields:

project:
  ShortName: "2025_Chickpea"
  FullName: ""
  description: ""
  start_date: ""
  end_date: ""
  funding_source: ""
  status: ""
  ProjectCode: ""
  Internal: null
  researcher:
    FirstName: ""
    LastName: ""
    Title: ""
    email: ""
    institution: ""
    role: "Principal Investigator"
    orcid: ""
  sites:
    - name: ""
      year: -9999
      season: ""
      SubLocation: ""
      latitude: .nan
      longitude: .nan
      description: ""
      ControlledEnvironment: null
      sensors: []

FieldLog.csv

Tracks individual field data collection events:

Column	Type	Description
Year	int	Year of collection
Month	int	Month of collection
Day	int	Day of collection
Sensor	str	Sensor platform name
Technician	str	Name of technician
Runs	int	Number of runs (≥ 1)
Site	str	Site name (must match ProjectSummary.yaml)
MakeNotesFile	bool	Whether to create a FieldNotes.txt
CheckSum	float	Integrity checksum (auto-computed)

Validation

The Rowchecker function validates every field log entry:

• Data types: Year, Month, Day, Runs must be int; Sensor, Technician, Site must be str
• Date validity: Must parse to a valid date, cannot be in the future, and must be within 14 days (unless --historical)
• Sensor match: Must be a sensor marked TRUE in the project summary CSV
• Run count: Must be ≥ 1
• Site match: Must exist in the ProjectSummary.yaml sites list with a matching year
• Checksum: Ensures row integrity hasn't been corrupted

Key Functions

Function	Purpose
main()	Entry point — orchestrates the full workflow
NodeChecker()	Ensures node folder and project summary CSV exist
projBuilder()	Loads project info, creates folders and metadata
Sitebuilder()	Creates sensor/date/run/tier folders for a field day
Rowchecker()	Validates a field log row
_projYAML()	Creates or loads project YAML
_sitenamemaker()	Generates standardised site folder names
_df_col_check()	Ensures DataFrames have required columns
GitPull()	Pulls latest changes from remote
GitChanged()	Checks and stages modified files