[1pt] PR: Copy files to scratch directory for processing, and remove upon completion

[robgpita]

In order to avoid file system collisions on AWS (keep the reads from the same file on disk to a minimum) four files (HUC6_dem_domain.gpkg, nws_lid.gpkg, reformat_ras_rating_curve_points_rel_101.gpkg, & usgs_gages.gpkg) are now copied from the inputs directory into a scratch directory (temporary working directory), and removed after processing steps are completed.

Resolves #1043 - (Merged dependabot/pip/jupyter-server-2.11.2 branch and tested)

Changes

• config/deny_unit.lst: Add files to remove list - repetitive copies needed for processing step (run_unit_wb.sh)
• src
  • bash_variables.env: Add a new variable for the ras rating curve filename. It will be easier to track the filename in the .env, and pull into run_unit_wb.sh, rather than hardcode it.
  • run_unit_wb.sh: Copy files and update references from $inputsDir to $tempHucDataDir.

Testing

Merged in PR #1043, built new docker container.

Ran fim_pipeline using new container and this branch.

No difference in hashes, using tools/hash_compare.py with outputs from 05030104 using dev branch, and this branch.

Ran BED Domain on AWS, using the Step Function Workflow successfully.

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• Passes all unit tests locally (inside interactive Docker container, at /foss_fim/, run: pytest unit_tests/)
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Reviewers requested
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[RobHanna-NOAA]

Looks good. Did a test of building a new image and running a small huc via pipeline and all is well.