admin.py: handle errors in getdirsize

[joshmoore]

see:

• https://latest-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/197/console
• https://py3-ci.openmicroscopy.org/jenkins/job/OMERO-test-integration/80/console

Both failed with:

01:24:20 OMERO data dir:'/home/omero/workspace/OMERO-test-integration/data' Exists? True	Is writable? True
01:24:21 Traceback (most recent call last):
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/bin/omero", line 131, in <module>
01:24:21     rv = omero.cli.argv()
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/cli.py", line 1756, in argv
01:24:21     cli.invoke(args[1:])
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/cli.py", line 1187, in invoke
01:24:21     stop = self.onecmd(line, previous_args)
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/cli.py", line 1264, in onecmd
01:24:21     self.execute(line, previous_args)
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/cli.py", line 1346, in execute
01:24:21     args.func(args)
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/install/windows_warning.py", line 26, in wrapper
01:24:21     return func(self, *args, **kwargs)
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/plugins/prefs.py", line 78, in open_and_close_config
01:24:21     return func(*args, **kwargs)
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/plugins/admin.py", line 1475, in diagnostics
01:24:21     dir_size = self.getdirsize(omero_temp_dir)
01:24:21   File "/home/omero/workspace/OMERO-test-integration/src/dist/lib/python/omero/plugins/admin.py", line 1555, in getdirsize
01:24:21     total += os.path.getsize(os.path.join(values[0], filename))
01:24:21   File "/home/omero/workspace/OMERO-test-integration/omero-virtualenv/lib64/python2.7/genericpath.py", line 49, in getsize
01:24:21     return os.stat(filename).st_size
01:24:21 OSError: [Errno 2] No such file or directory: '/home/omero/omero/tmp/omero_omero/6014'

Another process created a temp directory at the beginning of the run, but finished and was removed before getdirsize had completed.

[sbesson]

Overall the logic makes sense and I am surprised we have never encountered this scenario before on the shared devspace environment given the concurrency of the jobs. Off-hand I am assuming a race condition with the training job.

In the context of the temporary directory size, I think assumingdirectories might disappear during the calculation is safe and the code should prevent hard-failures. I do not see any other internal usage (https://github.com/search?q=org%3Aome+getdirsize&type=Code). Only question is whether we would like to be conservative for third-parties, implement a backwards-compatible implementation with some fails_on_missing=True flag that could be overriden by admin diagnostics.

[joshmoore]

Pushed a strict flag.

[sbesson]

Assuming tomorrow's builds/deployment pass, proposing to get this merged and monitor the absence of such failures in the upcoming weeks.