cdat_info extreme thread creation/CPU load needs work

[durack1]

As noted in CDAT/cdms#248 and CDAT/cdms#264 the thread (50+ processes spawned during 2 mins of a running script, with more if it runs for longer CDAT/cdms#248) and CPU load (4300% CDAT/cdms#264) for cdat_info when "log_anonymously": true is causing real problems, and is hitting per user thread limits (1024 on RHEL6) leading to process crashes due to thread blocks.

It would be great to get this thread and CPU load under control

@doutriaux1 @dnadeau4 @zshaheen

The following config in a script solves the problems

import os
# Must be done before any CDAT library is called.
if 'UVCDAT_ANONYMOUS_LOG' not in os.environ:
    os.environ['UVCDAT_ANONYMOUS_LOG'] = 'no'
# Used by numpy, causes too many threads to spawn otherwise.

But rather the software should just work out of the box, and not require custom env vars to be set by unsuspecting users

[durack1]

For other users this issue causes the following error when trying to login to a machine:

duro@ocean's password: 
shell request failed on channel 0

And for existing terminals the other familiar error, any time a new command is attempted:

"-bash: fork: retry: Resource temporarily unavailable" 

[durack1]

Copied from CDAT/cdms#248 (comment) @dnadeau4

@zshaheen note that if you say yes/no once, the file ~/.uvcdat/.anonymouslog is created. That file superseded your environmental variable until it is deleted.

This behaviour really needs to change

[durack1]

@doutriaux1 this should get you the behaviour, if you toggle threads on in top (Shift-H) you should see this behaviour

threadsTest.py.txt

[duro@ocean ~]$ more .uvcdat/.anonymouslog
{
  "log_anonymously": true, 
  "last_version_check": [
    "", 
    "8.0"
  ], 
  "last_time_checked": 1534367252.604961
}

[doutriaux1]

@durack1 I cannot reproduce this, when you are back please stop by my office.

[durack1]

@doutriaux1 @muryanto1 is this now in a nightly?

[muryanto1]

It should be in -c cdat/label/nightly now.

[doutriaux1]

thanks @muryanto1

[durack1]

So linux-64/cdat_info-8.0.2018.08.29.16.10.g300ba20-py27_0.tar.bz2 should include this?

[doutriaux1]

yep

[durack1]

@doutriaux1 I can confirm this fixes me, I was prompted to set the anonymous log info again, and set this as YES and this still seemed to work thread-free, thanks