Thread failure "too many open files"

[mzelinka]

Exception in thread Thread-10423:
Traceback (most recent call last):
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 277, in submitPing
    clean_cache()
  File "/usr/local/anaconda2/envs/2.10/lib/python2.7/site-packages/cdat_info/cdat_info.py", line 272, in clean_cache
    with bz2.BZ2File(cache_file,"w") as f:
IOError: [Errno 24] Too many open files: '/work/zelinka1/.uvcdat/.cdat_cache'

[doutriaux1]

ouch... I'll take a look right away. What were you doing?

[mzelinka]

Mutli-linear regression, but using numpy. It is not clear exactly where in the code the error is being generated.

[doutriaux1]

do you mind sending me the python code? By email if you don't feel comfortable posting it here.

[mzelinka]

I don't get the error when I switch back to cdat version 2.8

[doutriaux1]

yes this bz2 cache thing as been introduced in 2.10

[mzelinka]

Just curious if there are any updates on this? It is preventing me from using the latest version!

[doutriaux1]

I'll take a look either today or tomorrow.

[durack1]

@mzelinka @doutriaux1 @dnadeau4 @pochedls just wondering what happened with this back in Feb?

[pochedls]

I have some code in cdat82 that is producing this error (on f = cdms2.open). I believe I am closing all the file handles I open (after reading in the data).

[durack1]

@pochedls the cdat_info reference means that some weird logging thing with CDAT is likely causing your issue, if you make sure to set logging to off then this might fix you

[durack1]

@muryanto1 @downiec pinging you both as this is a nasty little bug that would benefit from a fix

[pochedls]

Does UVCDAT_ANONYMOUS_LOG='no' in my bash profile still turn off logging?

[durack1]

@pochedls theoretically yes

[doutriaux1]

@muryanto1 @downiec Dean really wanted the logging for statisictics purposes. But the server is pretty much broken (and might down Chris Mauzey would know) so it's probably safe to remove this. Double check with Ghaleb though. At least turn it off by default for now.

[downiec]

@doutriaux1 Yes, we discussed this last week actually and Ghaleb agreed we should go ahead and remove the logging at this time. I'm looking into removing it

[durack1]

@downiec to be honest I have hit my head against this many times, so removing it altogether is going to prevent a bunch of weird behaviors that slow down or make code bomb - two thumbs up for trim down

[github-actions]

Marking issue as stale, since there has been no activity in 30 days.

Unless the issue is updated or the 'stale' tag is removed, this issue will be closed in 7 days.

[durack1]

I believe this was resolved in the 8.2.1 release by #2 - so a candidate for closure

@downiec

[jasonb5]

@durack1 That's correct anonymous logging was removed, seems like the culprit. Closing as this should not be fixed.