Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1pt] PR: Add checks for intermediate files in AGREE #1103

Merged
merged 7 commits into from
Apr 17, 2024

Conversation

mluck
Copy link
Contributor

@mluck mluck commented Mar 22, 2024

Adds checks for intermediate files produced by Whitebox in the AGREE process (src/agreedem.py). Without these checks, if Whitebox fails to produce an output, no error is generated until much later in the src/delineate_hydros_and_produce_HAND.sh processing chain which makes troubleshooting difficult. Closes #1102.

Changes

  • src/agreedem.py: Added checks to verify existence of intermediate files before continuing

Testing

Correctly identified missing file. For example, agreedem.py now exits with an AssertionError rather than continuing:

Creating AGREE DEM using 70 meter buffer 18060004 0
Traceback (most recent call last):
  File "/foss_fim/src/agreedem.py", line 307, in <module>
    agreedem(
  File "/foss_fim/src/agreedem.py", line 183, in agreedem
    assert os.path.exists(bufallo_grid), f'Buffer allocation grid not created: {bufallo_grid}'
AssertionError: Buffer allocation grid not created: /fim_temp/18060004_test/18060004/branches/0/agree_bufgrid_allo.tif
Command exited with non-zero status 1
	Command being timed: "/foss_fim/src/run_unit_wb.sh"
	User time (seconds): 359.59
	System time (seconds): 46.01
	Percent of CPU this job got: 96%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 7:01.58
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 2692564
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 46969
	Minor (reclaiming a frame) page faults: 10277735
	Voluntary context switches: 59846
	Involuntary context switches: 45217
	Swaps: 0
	File system inputs: 9464744
	File system outputs: 3127312
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 1

***** An error has occured  *****

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • Passes all unit tests locally (inside interactive Docker container, at /foss_fim/, run: pytest unit_tests/)
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Reviewers requested
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@RobHanna-NOAA RobHanna-NOAA self-requested a review April 15, 2024 19:32
RobHanna-NOAA
RobHanna-NOAA previously approved these changes Apr 16, 2024
@CarsonPruitt-NOAA CarsonPruitt-NOAA merged commit 729cc01 into dev Apr 17, 2024
1 check passed
@CarsonPruitt-NOAA CarsonPruitt-NOAA deleted the dev-check-agree-intermediate branch April 17, 2024 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[1pt] Missing intermediate file in AGREE processing causing error
3 participants