Call for Feedback on Repository

[Schiano-NOAA]

🧾Background

In an effort to increase throughput and streamline stock assessment workflows, a team at NOAA Fisheries has been working on a potential end-to-end stock assessment pipeline that links many new and current efforts together. One piece of this pipeline is a standard repository for a stock assessment that would house all of the necessary files to run a stock assessment from data processing for the input file to building the stock assessment report. The intention of this repository is to be set up one time and maintained over time. This is a template repository meaning that the user copies it from the "Use this template" button in GitHub (similar to forking). The initial creation of the repository should replace the region and species for the stock in the title followed by stock assessment. Year is not an indicator in the repository title/name. This is intentional. Each successive assessment cycle uses the same repo for the region and stock, but uses the tagging function in GitHub to identify what scripts and files were current at the end of the process during that year.

The thoughts behind crafting this repository are:

• Information of the README containing instructions will be replaced with some other template for navigating the repository for the stock
• Each folder separates and organizes the scripts made in the process
• Data is not housed in GitHub (this includes model output files)
• (Potentially) R functions are stores in the R folder so the analyst can run devtools::load_all() to access their functions in a more efficient manner
• The template contains files to help you set up a workflow in the cloud and other automated processes
• Each folder contains a README file describing what types of files go there

👍 Gathering Feedback

We are looking to gather feedback on this template repository. We are hoping the stock assessment community and others involved in the process can look over the structure and organization of this repository and provide details on what you like about it and how you would improve it.

Please indicate what region you are from and what your role is. This will help us gain insights into how each region (or person) approaches their organization of files. Feel free to also add in suggestions or mention anything that might be missing from this template.

Thank you for your participation!! 🥳

[grantdadams]

Awesome! Standardized assessment folders and reproducible assessments are a dream.

Looking at the structure, I think people (and I) will be confused on the difference between the R and data-raw (based on the read-me) folders. My expectation would be that the data-raw folder includes the pre-processed data, the data folder includes the post-processed data ready for input into the assessment model, and the R folder includes all the functions, and scripts has the scripts to run processing of the data, the model, and diagnostics/visualization.

I think it will also be important to have:

1. A folder for plots and tables that may or may not be in the final report (e.g. r4ss outputs a lot of plots)
2. A folder(s) for the final model and model sensitivity files (likely deprecated as people move away from ADMB)

[Schiano-NOAA]

Thanks Grant! The suggestion for folders is a good thought.

I think for the folder of the final model and model sensitivity, I would like to hear others weigh in on this as we had the thought! Ideally, output gets stored in long term storage whereas the rest of the model files can be reproduced through the code and content found in the repo.

[brianlangseth-NOAA]

I would recommend a generic models folder over just a single specific to the final base model. Like Chantel's structure below, any sensitivities, etc could lie within the generic models folder. The details of that specificity for subfolder is likely best to leave up to the user.

[BenWilliams-NOAA]

While I appreciate the objective of this type of template I have some misgivings of attempting to prescribe how assessment repos are setup. I've come to this conclusion from reviewing and being involved with a number of assessments. There is such a wide variety of inputs/outputs, necessary components etc. that they effectively, or at least efficiently, cannot be covered. For example, for some years this will be straightforward e.g., update assessment. But for a full assessment, you get into research track, mgmt track, iterative model changes, etc. A current working-ish example is

    year 1  
    year 2
      | -- R
      | -- data
      |     |--raw
      |     |--user_input
      |     |--output
      |     |--sara
      |     |--sql
      | -- mgmt
      |     |--m26.0
      |     |--m26.1
      | -- research
      |     |--m26.1a
      |     |--m26.1b
      | -- plan_team
      | -- safe

Though this setup won't work for every scenario and probably won't work for some assessment software.

How does this template account for years? Having separate R and scripts folders is a bit at odds with standard R setup (think packages). Additionally, most of the functions to run data pulling/cleaning, generate figures etc. will be sourced (or better in an R package themselves) as they don't really change that often - hopefully. Seems a bit more logical to simply have a R folder.

I think that within the context of asar and stockplotr it is worth considering simply stating that a project needs to have x components or folders in place in order to run effectively. This makes a uniform input/output for the functions to run but doesn't prescribe the overall structure for wildly varying assessments.

[kellijohnson-NOAA]

@BenWilliams-NOAA can you elaborate on how having separate folders for R and scripts is against standard R setup? In the R folder there would only be "functions" so load_all() would put all functions in your R environment. Whereas scripts would be actionable R code that potentially uses functions found in the R folder.

[BenWilliams-NOAA]

well - all of my "scripts" are in an R package(s). So would not use load_all() nor need a scripts folder. Was more thinking in terms of an R package - which imo is a good setup for a stock assessment - minus the "live" document aspect of an assessment with continually updating data. An R package has defined components with clear meaning and documentation. We've been chatting about this for many years at AFSC, as I'm sure many other regions have as well - and we haven't landed on an option that works for all.

Oh, per your naming conventions comments. I'm a fan of using shorter clear names and then using "Topics"/"Tags" to filter.
For instance one can filter based on "groundfish"

If you look at goa_nork on the right side there are a number of topics that can be filtered in lieu of a long repo name.

[Schiano-NOAA]

Thanks Ben for your response! In regards to your question about years, the thought is that you would use the tags/releases feature of GitHub to save a point in time which the assessment is done and tag it for that year. On the next cycle, you would use the same repo and overwrite/update the necessary places.

[BenWilliams-NOAA]

I'm cautiously optimistic that the use of tags/releases is within the general git/github skillset of most of our git users. There would certainly need to be more training on exactly how/when to do so in order to avoid pitfalls like overwriting a tag and having git's garbage collection throw out the old. Worth considering anyway. I'm a huge proponent of standardized data/code structures, but am also aware that developers can spend a lot of time directing outputs to certain locations when ultimately it would be efficient to simply provide recommendations of how/where to store items and let people do what they may. Playing a bit of devil's advocate as there is a lot to think through with this type of venture.

[brianlangseth-NOAA]

I also was a bit confused over the separate R versus scripts folders though Kelli's explanation above clarifies that to some degree.

[kellijohnson-NOAA]

I think the repository name should start with "stock_assessment" then have the adjectives for the species at hand, e.g., "stock_assessment_lingcod_north", that way all repositories will be listed close together within an organization because each one will start with "stock_assessment_". Second, the "region" qualifier should be last. Because in the future, a population might be split between regions or grouped across regions, e.g., sardine. This way the repositories for a given species will all be similarly named with only the trailing words varying.

Another idea but definitely not a have to: We could shorten "stock_assessment_" to be "SA_" so it isn't so long to type.

[Schiano-NOAA]

In the past, I think we have discussed using tags for specific repositories (what ben mentioned too). Maybe that's something we could do here as well.

[AdyanRios-NOAA]

I've had my own structure for our Caribbean assessments that isn’t all that different, though I’ve been creating entirely new repos for updates because I don't know how to do versioning yet. For example, my recent update repo for PR Caribbean Spiny Lobster (CSL) through 2024 is called SEFSC-SEDAR-Carib-91U-PR-CSL-2024. The name has our Region (SEFSC), notes it is a SEDAR, has our subregion (Carib), notes the SEDAR # and if it is an update (91U)... followed by the Island Platform, species short name and terminal year

For context, my region is the Southeast (SEFSC/Caribbean) and my role is a stock assessment analyst.

My root directory looks like this:

• code/
• data/
• figures/
  • specific_comparison/
• quarto/
• scenarios/
  • specific_run/
    • scenario_inputs/
    • plots/
    • diagnostics/
      • retrospective/
      • profile_specific_par/
      • jitter/

When I started working with Jason Cope's SAC a bit, I converged on having all my runs in a scenarios folder, with one folder per run. I write run-specific plots and diagnostics within those individual run folders, but save comparison plots across runs to the main figures folder.

Most data-limited models in the Caribbean don't end up with just a single run. To keep the repo clean-ish, I have my gitignore set up to only push the four main SS files for each specific run, plus my R code/ and quarto/ files. I keep a script in the code folder that reruns all models, plots, and diagnostics (or just specific runs) using {foreach} to theoretically get back up running from a fresh clone. Note, Caribbean models don't take long to run.

One detail to watch out for with the folder structure is that my setup works great for Quarto PDFs, but it breaks for HTML because the figures and tables end up too nested. I'm not sure if that’s been fixed yet, but I just stuck with PDFs because I definitely didn’t want to copy & move files or have append run names to all similarly named figure files into a figures/ folder at the root. If the template is going to eventually also have nice HTML easily posted as GitHub pages, it's worth double-checking how it handles deeply nested relative paths.

Thanks for putting this together and reaching out!

[AdyanRios-NOAA]

When the time comes, I think it will be relatively straightforward for me to transition more into asar's structure. Just wanted to share those notes about multiple model runs and how I am currently set up.

[Schiano-NOAA]

Thanks so much Adyan! 😄 This is helpful information. I think it is sounding like we will need to expand on organization of model files within this setup, but I think most regions will need to branch from here within each folder. The idea is that people are able to point to and locate files within any general workflow and navigate a bit better. This could help with automated approaches in the (possibly near) future.

[chantelwetzel-noaa]

I really like the proposed structure. I set up my most recent assessment in a similar fashion and found that it worked really well. I support trying to create more standardization around how we organize our work, at least around the key items. I know there will need to be additional folders for other items associated with an assessment that may not fit the prescribed folder structure, and I think that is fine.

Here is how I structured my most recent assessment:

├───data (rda files created by scripts in data-raw)
├───data-raw (R scripts that call functions to create processed data for the assessment)
├───date-exploration (R code used to understand the data)
├───data-tables (processed csv files)
├───R (functions used to process data)
├───man
└───report 
│   └───figures
│   └───SAR_USWC_sablefish_skeleton_files (git ignored)
├───quarto_website
├───model (git ignored)
│   └───base_model
│     └─── folder with the current base model
│     └───sensitivities
└───presentations (revealjs files)

[jimianelli]

I've been working on updating the workflow for my assessment and have experimented with several options. I tried adopting your setup and came up with:

https://github.com/afsc-assessments/EBS_pollock_stock_assessment

With obvious "help" I tested the template concept using the 2024 EBS pollock assessment. The basic folder structure is useful. The additions that seemed most important for the generic template were a complete runnable example, clarification of whether this is an R package or project, guidance for organizing multiple assessment years, guidance for external or restricted data, dependency management, and a simple automated workflow test. Alaska assessments require additional conventions for model comparisons, diagnostics, projections, and review stages, but those are probably better demonstrated in regional examples than required by the general template.

In summary, the basic R/, data-raw/, data/, scripts/, and report/ structure works well, but the exercise suggested that operational assessments may also benefit from:

• a separate folder for each assessment year;
• tracking tables showing where each input came from and where each final product is stored;
• a clear record of how the base model, bridge runs, alternatives, sensitivities, diagnostics, and projections relate to one another;
• a documented connection between the final assessment model and the projections used to calculate harvest recommendations;
• a record of which model and results were recommended by the authors, Plan Team, and SSC, and which were ultimately accepted;
• instructions for accessing confidential or very large files that cannot be stored in the repository;
• a record of the software, packages, model versions, and other dependencies needed to reproduce the work;
• automatic checks that required tracking information is present and that the assessment report builds successfully;
• Git tags identifying important annual milestones, such as the authors’ submission, Plan Team version, and final accepted assessment.

Some of these are obvious things that probably don't need to be in a template; the list mixes

• basic repository organization;
• Alaska-specific assessment practices;
• details that belong in a completed assessment, not the generic template.

The OST template should probably remain lightweight. It does not need prescribed fields for Plan Team and SSC review, every diagnostic type, harvest projections, or milestone tags. Those conventions vary among regions and assessments.

Potential gaps that affect whether the template works as advertised might be to:

• Include one small, complete example showing how data preparation, model execution, and report generation connect.
• Clarify whether the repository is intended to be an R package or an ordinary R project. Currently it recommends devtools::load_all() without providing package infrastructure.
• Provide a simple convention for organizing multiple assessment years without duplicating the entire repository.
• Explain how to document data that cannot be committed because it is confidential or too large.
• Add an example dependency record, such as renv.lock, and one documented command that runs the workflow.
• Add a basic automated check that the example runs and the report renders.
• Replace the plaintext GitHub-token instructions with safer credential guidance.

[jimianelli]

and I did this stupidly before reading the other comments above. So maybe my "year" parent folder isn't such a great idea. Seemed useful since it might help with organizing bridging over the furlough year (2025...). I think the strongest follow-up is to acknowledge that others have already addressed the "years" issue and pivot toward what you learned from actually trying the template. That adds something new to the discussion rather than defending your earlier suggestion.

The exercise of converting the 2024 EBS pollock assessment into the template was still valuable because it highlighted a few things that seem broadly useful regardless of how years are organized:

• A small, complete runnable example would make it much easier to understand how the pieces fit together.
• The repository should be explicit about whether it is intended to be an R package or simply an R project, especially if devtools::load_all() is part of the recommended workflow.
• Some guidance is needed for documenting external, confidential, or very large datasets that can't live in GitHub.
• Reproducibility would benefit from documenting dependencies (e.g., renv.lock) and including a simple automated check that the example builds successfully.

The Alaska assessments also reminded me that there are many region-specific conventions (bridge runs, sensitivities, projections, review stages, etc.). Those seem better handled as regional examples built on top of a lightweight common template rather than requirements of the template itself.

Overall, I still think the basic folder organization is a solid starting point—the value is having enough consistency that someone new can quickly navigate an assessment repository without trying to standardize every region's workflow.

[brianlangseth-NOAA]

I wonder whether folders for separate years are necessary. We've debated the pros and cons of using the same repository each cycle and just writing over everything and managing version control with tags/releases, or using different repositories. Using separate folders allows a lot of the discussions/issues to remain (which is a benefit over different repositories), but would seem to duplicate a lot of the same material. I have had one situation where I did another assessment for the same species, but couldn't get myself to come around to the idea of starting again within the same repository. Would be curious to hear how else others handle this issue (and based on the discussion so far, it looks like year folders looks like the most common).

[iantaylor-NOAA]

I'll be happy to have a standard template and can work with whatever @Schiano-NOAA and the team come up with.

Regarding the years issue, here's an option that has worked reasonably well for me: using a model naming convention that includes the year in the folder name so that all model folders can be just nested directly under "models". For example, in this assessment repo for the north and south lingcod assessments https://github.com/pfmc-assessments/lingcod/tree/main/models which @kellijohnson-NOAA and I created in 2021, we pushed

• 3 models from 2017 (e.g. 2017.n.001.001.final_base),
• 10 models from 2019, and
• 162 models for 2021.

When lingcod is assessed in 2027, it seems like a reasonable approach would be to remove the majority of the 2021 models, leaving only the base model and a few other important alternative models or sensitivities while adding new 2027 models. If the old models are filtered early in the assessment process, it's possible that we would over- or under- filter, but it would be easy to refine that going forward, either adding back models from the tagged 2021 version of the repo or deleting those which aren't valuable any more. Aside from the models, it's hard for me to imagine that it's important to retain R code or document text from a previous assessment year in the active repo as opposed to the git history.

I should add the caveat that my perspective comes from west coast groundfish perspective where most stocks are assessed infrequently, so we don't have that much experience updating assessment github repos. And it also seems like a generic OST template could be designed to allow users to use either year-specific folders or include the year in the model name to cover a range of preferences.

[jimianelli]

wow das a lot of model runs. I started to clone it to get a better feel but it seemed rather large so cancelled it. Seems really useful for the 2027 person doing the assessment to have it all there. AI will be super helpful to them I imagine in distilling things down and focusing on the next big thing.

[iantaylor-NOAA]

I was thinking that one way to "have it all there", as Jim notes, would be to create a separate clone of the repo at the tag associated with the previous repo (2021 in this case) which would be parallel to the filtered version on main focused on 2027. However, it looks like a better approach than separate clones is worktrees: https://github.blog/ai-and-ml/github-copilot/what-are-git-worktrees-and-why-should-i-use-them/. Either way, I think it's possible to reduce the bloat of the visible files in the github repo (maybe the git history would still get too bloated).

[kellijohnson-NOAA]

With the template, we are operating under the thought that the model results would not be pushed to GitHub. Instead we are working with headquarters on ways to nationally archive model results, including sensitivity runs, so you can use an API to access last year's model and temporarily put current results in cloud buckets to allow multiple users to access them while building the assessment.

[cmlegault]

How future-proof is the suggested approach of using tags/releases/worktrees within GitHub? Having seen many changes to operating systems, software, stock assessment approaches, etc. over the course of my career, I worry this solution may put everyone in a hard spot if some future owner of GitHub decides to remove (or charge exorbitant prices for) tags/releases/worktrees.

[iantaylor-NOAA]

@cmlegault, good thing to think about, but I think the danger is low. The underlying git software has been in place since 2005 is GPL licensed and I can't imagine it going anywhere anytime soon because it's so widely used and can be run locally by anyone. Tags and worktrees are features of git. Releases are really just a collection of files, including a zip file of the code at that snapshot in time. NOAA policy requires us to have backups for these things (see https://nmfs-opensci.github.io/GitHub-Guide/#sec-backups) and I like the in-development idea of a national archive of assessment results that @kellijohnson-NOAA mentioned last week which seems better than the current combination we have at NWFSC center of local laptops + shared drive archive on Google Drive.

GitHub is owned by Microsoft and could certainty change in some way in the future that no longer works for us, but if that happens transferring all of our stuff elsewhere isn't a big deal. When I first put the r4ss package online in 2013, it was using the svn version control system instead of git and it was on Google Code, which was shut down in 2016. Transferring the full commit history and issues to git and GitHub wasn't a big deal. Likewise Stock Synthesis first went into version control on the NOAA VLab and @k-doering-NOAA later transferred the full commit history and all the active issues over to GitHub without too much trouble.

[Schiano-NOAA]

Thank you all for your feedback so far! I have been keeping an eye on it and as it has slowed down, I will take into consideration your suggestions and make edits to this repo. I will also be getting back to you all regarding some of your thoughts. This has been such an an engaging conversation!!