Adding statewide monitoring workflow under data.remote inst/ccmmf by sarahkanee · Pull Request #3913 · PecanProject/pecan

sarahkanee · 2026-03-31T20:38:37Z

Includes hls, phenology, events, tillage, traits.
Single tree for dependent scripts (events + pool + tillage). Should break this up at some point

Description

Motivation and Context

Review Time Estimate

Immediately
Within one week
When possible

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
My name is in the list of CITATION.cff
I agree that PEcAn Project may distribute my contribution under any or all of
- the same license as the existing code,
- and/or the BSD 3-clause license.
I have updated the CHANGELOG.md.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Includes hls, phenology, events, tillage, traits. Single tree for dependent scripts (events + pool + tillage). Should break this up at some point

…toring-inst

Includes orchestrator, crop/subclass and ADOY gap-fill, product build, QC reporting, and SGE submission scripts. Excludes large runtime artifacts (CDL fractions, trained parquet tables, logs).

Bring lab management code (extract packages, events/phenology/hls/traits, stakeholder docs, Session 0 env template) onto the PR branch. Ship county crop matrices as real files and resolve shared HLS libs via sibling paths so clones work without /projectnb defaults.

Session 1 adds an operator runbook for cadwr-landuse (01-03) while pointing at Alexey's README/docs as canonical; pipeline and Session 0 use user wording and ASCII.

Session 1 adds a CLASS/SUBCLASS lookup check (ignore DWR ** sentinels), spells out management/*-with-YEAR publish dirs, and checks out the cadwr-landuse auto-discover branch until main has it.

These one-off modislc/skanee/neochatt path scripts are lab ops, not part of the portable monitoring stack shown in PR PecanProject#3913.

Sessions 0-4 and pipeline are the curriculum; package READMEs are detail docs linked from sessions, not a parallel TOC.

Lead README, Session 0/1, and pipeline with what the trainee does (add 2024, re-run 2023), portable paths first, and a clearer checklist.

mdietze · 2026-07-23T00:05:24Z

+**You are here to process a new LandIQ year.**
+
+This training walkthrough adds **2024** and re-runs **2023** (the year pair).
+You work on **your own Linux cloud / HPC** (not BU SCC).


This statement is only meaningful to people who ARE working on the BU SCC. No one else will know what that is or consider that the "default"

mdietze · 2026-07-23T00:06:47Z

@@ -0,0 +1,56 @@
+# CCMMF monitoring pipeline - documentation
+
+**You are here to process a new LandIQ year.**


What's LandIQ? what's a LandIQ year? Why would I want to process one and what does that mean?

mdietze · 2026-07-23T00:09:31Z

@@ -0,0 +1,34 @@
+# CCMMF statewide monitoring (`inst/ccmmf`)
+
+California LandIQ → phenology → management-event pipeline scripts for PEcAn.


If I was looking in this folder for the first time, how would I know what this is? You need to start more broadly and explain yourself. Also, I don't think it's fair to assume that the ONLY thing you'll end up putting in the data.remote/insts/ccmmf folder will be phenology

mdietze · 2026-07-23T00:09:53Z

+
+**Documentation (start here):** [documentation/README.md](documentation/README.md)  
+**Environment setup:** [documentation/sessions/00-environment.md](documentation/sessions/00-environment.md)  
+**PR:** https://github.com/PecanProject/pecan/pull/3913


not sure why the documentation would reference a PR. Drop

mdietze · 2026-07-23T00:12:04Z

+
+| Path | Role |
+|------|------|
+| `documentation/` | Stakeholder sessions + `pipeline.md` + `ccmmf_env.example.sh` |


Expand every single "role". Even knowing how this workflow works, I find the descriptions overly terse and cryptic. Any time you read/write documentation or tutorials, you have to put yourself in the shoes of someone who has never heard of what you're doing and is looking at this project for the first time

mdietze · 2026-07-23T00:31:58Z

+export CCMMF_ROOT=/projectnb/dietzelab/ccmmf
+
+export LANDIQ_GAPFILL_ROOT=$CCMMF_ROOT/management/landiq-gapfill
+export CCMMF_LANDIQ_V4=$CCMMF_ROOT/LandIQ-harmonized-v4.1


I don't think it's safe to define variable names with version numbers in them, it will require a lot of code fixes (and potential bugs) when LandIQ jumps to V5.

mdietze · 2026-07-23T00:32:52Z

+A routine run updates only the years you pass and carries the rest over from the
+existing product, so adding a year is cheap.
+
+On a routine run, emission tables should **not** rebuild. If they do, confirm


what are emission tables?

mdietze · 2026-07-23T00:34:07Z

+compute node. Run the whole thing as one batch job:
+
+```bash
+qsub -l buyin -l h_rt=8:00:00 -v 'GAPFILL_ARGS=2023,2024' $LANDIQ_GAPFILL_ROOT/sge/run_gapfill.sge


this won't run for other people. You need to do more to explain concepts so users can understand what something is supposed to do. otherwise they won't know how to configure it for their own use case

mdietze · 2026-07-23T00:35:28Z

@@ -0,0 +1,32 @@
+#!/bin/bash -l
+#
+# SGE job: download CDL GeoTIFF(s) from NASS CropScape (avoids login-node timeout).


unclear what SGE is

these scripts look like they will only run on the BU SCC, not CARB's server or any else's

mdietze · 2026-07-23T00:38:17Z

+})
+
+.fa <- sub("^--file=", "", grep("^--file=", commandArgs(trailingOnly = FALSE), value = TRUE)[1L])
+source(file.path(dirname(normalizePath(.fa, mustWork = FALSE)), "_lib", "pkg_root.R"))


as your code matures towards production quality, you should be using "source" less and relying on the package more. This will mean moving more things into the /R folder and also using system.file more than file.path

Sessions and package README now point at pipeline.md. Package README explains CCMMF broadly without a PR link and clarifies first-time setup from inst/ccmmf.

…sh docs. Fold MSLSP into phenology/extract and NDTI into tillage/extract; move tillage metrics into events; remove SGE/explore leftovers; rewrite pipeline and session docs around the year-pair training path.

…urb.

Removed first-time setup instructions from README.md and streamlined the content.

Added instructions for running the pipeline in the README.

Keep "package" only for real R/Python/PEcAn packages.

Removed duplicate instruction for running the pipeline.

Updated descriptions for various paths and removed first-time setup instructions.

Removed redundant line about following pipeline documentation.

Updated section heading from 'tree' to 'branch' for clarity.

Removed reference to LandIQ crop code lookup table from README.

Removed reference to cdl_fractions_metadata.csv in the documentation.

Updated the algorithm section to clarify the process of joining NDTI scenes and determining tillage dates.

Removed reference to older flat imagery trees and associated README.

Updated contact reference for session lead in documentation.

Adding statewide monitoring workflow under data.remote inst/ccmmf

aa12ea4

Includes hls, phenology, events, tillage, traits. Single tree for dependent scripts (events + pool + tillage). Should break this up at some point

github-actions Bot added the modules label Mar 31, 2026

mdietze and others added 9 commits April 17, 2026 14:22

Merge branch 'develop' into feature/ccmmf-statewide-monitoring-inst

f0fd483

Merge branch 'PecanProject:develop' into feature/ccmmf-statewide-moni…

ee61620

…toring-inst

Add LandIQ gap-fill workflow under data.remote inst/ccmmf

c3be9ef

Includes orchestrator, crop/subclass and ADOY gap-fill, product build, QC reporting, and SGE submission scripts. Excludes large runtime artifacts (CDL fractions, trained parquet tables, logs).

Document portable LandIQ year-add ops and two-repo user flow.

46e13d9

Session 1 adds an operator runbook for cadwr-landuse (01-03) while pointing at Alexey's README/docs as canonical; pipeline and Session 0 use user wording and ASCII.

Document legend QC recipe and writable SCC LandIQ product paths.

aa6468e

Session 1 adds a CLASS/SUBCLASS lookup check (ignore DWR ** sentinels), spells out management/*-with-YEAR publish dirs, and checks out the cadwr-landuse auto-discover branch until main has it.

Remove SCC-only HLS ancillary transfer scripts from inst/ccmmf.

dc07510

These one-off modislc/skanee/neochatt path scripts are lab ops, not part of the portable monitoring stack shown in PR PecanProject#3913.

Clarify documentation README as user training path.

4fe952b

Sessions 0-4 and pipeline are the curriculum; package READMEs are detail docs linked from sessions, not a parallel TOC.

Rewrite docs around the 2024 + 2023 user year-pair walkthrough.

98a032e

Lead README, Session 0/1, and pipeline with what the trainee does (add 2024, re-run 2023), portable paths first, and a clearer checklist.

mdietze reviewed Jul 23, 2026

View reviewed changes

sarahkanee and others added 18 commits July 23, 2026 10:58

Drop documentation/README; make pipeline.md the doc spine.

d616245

Sessions and package README now point at pipeline.md. Package README explains CCMMF broadly without a PR link and clarifies first-time setup from inst/ccmmf.

Restructure inst/ccmmf for training: fold extracts, drop attic, refre…

ab611f7

…sh docs. Fold MSLSP into phenology/extract and NDTI into tillage/extract; move tillage metrics into events; remove SGE/explore leftovers; rewrite pipeline and session docs around the year-pair training path.

Update metadata.md

a6cfee1

Update environment documentation

4d44a4e

Drop troubleshooting sections from ccmmf READMEs and trim root doc bl…

d6743d6

…urb.

title

8ff6702

title

0726eab

Update README.md

3e516da

Removed first-time setup instructions from README.md and streamlined the content.

Update README

dd49cf7

Added instructions for running the pipeline in the README.

Give inst/ccmmf README a formal monitoring-pipeline introduction.

8ea66fd

Point CCMMF README at pipeline.md instead of a docs index.

a401599

Refer to management inputs as management events in intro docs.

8204bc5

Call pipeline folders components, not packages.

687f389

Keep "package" only for real R/Python/PEcAn packages.

Update README.md

7f456e5

Removed duplicate instruction for running the pipeline.

Update README.md

02dc5ea

Updated descriptions for various paths and removed first-time setup instructions.

Update README.md

d6cc44a

Removed redundant line about following pipeline documentation.

Update README.md

6952bee

Updated section heading from 'tree' to 'branch' for clarity.

Remove LandIQ crop code lookup table reference

da412a0

Removed reference to LandIQ crop code lookup table from README.

sarahkanee added 9 commits July 24, 2026 14:23

Update 01-landiq.md

d09f7d2

Update 01-landiq.md

16e9a88

Update documentation for gap-fill process

6c1bbc3

Remove cdl_fractions_metadata.csv reference

6d4669b

Removed reference to cdl_fractions_metadata.csv in the documentation.

Update 02-phenology.md

62d1a6b

Clarify algorithm description in tillage documentation

abfdae1

Updated the algorithm section to clarify the process of joining NDTI scenes and determining tillage dates.

Update documentation for tillage fertilizer

e962b82

Removed reference to older flat imagery trees and associated README.

Change session lead reference to Akash

2be6e98

Updated contact reference for session lead in documentation.

Update CCMMF title in README.md

a9b7967

Navigation Menu

Uh oh!

Adding statewide monitoring workflow under data.remote inst/ccmmf - #3913

Adding statewide monitoring workflow under data.remote inst/ccmmf#3913
sarahkanee wants to merge 37 commits into
PecanProject:developfrom
sarahkanee:feature/ccmmf-statewide-monitoring-inst

sarahkanee commented Mar 31, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,56 @@
		# CCMMF monitoring pipeline - documentation

		You are here to process a new LandIQ year.

		@@ -0,0 +1,34 @@
		# CCMMF statewide monitoring (`inst/ccmmf`)

		California LandIQ → phenology → management-event pipeline scripts for PEcAn.

Uh oh!

Conversation

sarahkanee commented Mar 31, 2026

Description

Motivation and Context

Review Time Estimate

Types of changes

Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants