Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn about setting a GDAL cache max if probably on a SLURM node #361

Closed
phargogh opened this issue Jan 16, 2024 · 0 comments · Fixed by #362
Closed

Warn about setting a GDAL cache max if probably on a SLURM node #361

phargogh opened this issue Jan 16, 2024 · 0 comments · Fixed by #362
Labels
enhancement New feature or request

Comments

@phargogh
Copy link
Member

After some time away from Sherlock, I wanted to run a simple pygeoprocessing function call on a relatively small raster but found myself exhausting memory. Of course, I had forgotten that I was running this within a SLURM job, which may have some extra identifying features available, such as environment variables.

It would be really great to be able to, at import time, log a warning or something that we are running in a slurm job and that gdal's cache max is unset, defaulting to a very large memory limit. Even better would be if we could detect the slurm job memory limit and compare it to the GDAL cache limit, but even a simple warning would be great.

@phargogh phargogh added the enhancement New feature or request label Jan 16, 2024
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Jan 17, 2024
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Jan 18, 2024
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Jan 18, 2024
Pytest is elevating warnings to exceptions, so replacing the testing of
warnings capture is sufficient to get around this.

RE:natcap#361
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Jan 18, 2024
phargogh added a commit to phargogh/pygeoprocessing that referenced this issue Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant