Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

met2model.BIOCRO leaks filehandles when processing multiple years #2485

Closed
infotroph opened this issue Nov 27, 2019 · 0 comments · Fixed by #2486
Closed

met2model.BIOCRO leaks filehandles when processing multiple years #2485

infotroph opened this issue Nov 27, 2019 · 0 comments · Fixed by #2486
Milestone

Comments

@infotroph
Copy link
Member

infotroph commented Nov 27, 2019

Bug Description

Found while working on #2484: PEcAn.BIOCRO::met2model.BIOCRO loops over as many years of CDF inputs as you ask it to process, but only calls nc_close once at function exit. When processing more than one year, this results in the filehandles for all but the last year being left open.

Patch incoming, just documenting it here for posterity.

To Reproduce

Edit paths freely -- I'm showing with GFDL because it's what I used, but results should be the same with any multi-year data source you have handy.

In an R session that will stay alive after function exit (i.e. not just Rscript -e):

PEcAn.BIOCRO::met2model.BIOCRO(
  in.path="~/pecan/dbfiles/GFDL_site_0-753/GFDL_CM3_rcp45_r1i1p1_site_0-753/", 
  in.prefix="GFDL.CM3.rcp45.r1i1p1",
  outfolder="/tmp/gfdl_biocro_test/",
  lat=40,
  lon=-88,
  start_date="2009-01-01",
  end_date="2011-12-31",
  overwrite=TRUE)

In another terminal, with the above R session still active (so that we see the leaks before they're cleaned up by the OS when R exits):

lsof -c R -a +D ~/pecan/dbfiles/

Expected behavior

$ lsof -c R -a +D ~/pecan/dbfiles/
$

Observed behavior

$ lsof -c R -a +D ~/pecan/dbfiles/
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
R       97927 chrisb   24r   REG    1,4   118164 4363299343 /Users/chrisb/pecan/dbfiles/GFDL_site_0-753/GFDL_CM3_rcp45_r1i1p1_site_0-753/GFDL.CM3.rcp45.r1i1p1.2009.nc
R       97927 chrisb   26r   REG    1,4   118164 4363297178 /Users/chrisb/pecan/dbfiles/GFDL_site_0-753/GFDL_CM3_rcp45_r1i1p1_site_0-753/GFDL.CM3.rcp45.r1i1p1.2010.nc
$

Proposed fix

Explicitly call ncdf4::nc_close at the end of each year's processing. We should keep closing the file in on.exit to make sure we aren't leaking filehandles when the function terminates early, but will need to do it conditionally to avoid ncdf4 complaining when we try to close files that were closed already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants