Update supertask interface and many related classes. #17
Conversation
andy-slac
force-pushed
the
tickets/DM-13452
branch
from
f7f907c
to
a4c71d3
Compare
andy-slac
force-pushed
the
tickets/DM-13452
branch
3 times, most recently
from
62987ab
to
8472bd5
Compare
Still WIP, many updates to parser, SuperTask, SuperTaskConfig, unit test and examples.
andy-slac
force-pushed
the
tickets/DM-13452
branch
from
8472bd5
to
0440bd0
Compare
Re-enable unit tests for graph builder and supertask.
| @@ -307,7 +303,7 @@ def runPipeline(self, graph, butler, args): | |||
| mapFunc = _MPMap(numProc, timeout) | |||
| else: | |||
| # map in Py3 returns iterable and we want a complete result | |||
| mapFunc = lambda func, iterable: list(map(func, iterable)) | |||
| mapFunc = lambda func, iterable: list(map(func, iterable)) # noqa: E731 | |||
There was a problem hiding this comment.
I think we really should have a def mapFunc here rather than something which looks like python word salad (and we don't need the python2 code).
| @@ -346,13 +342,12 @@ def _executeSuperTask(self, target): | |||
| return task.runQuantum(quantum, butler) | |||
| except Exception: | |||
| raise # No worries | |||
| except: | |||
| except: # noqa: E722 | |||
There was a problem hiding this comment.
No. This should be BaseException if you are really wanting to intercept Ctrl-C and the like. I'm not sure that's a good idea though, especially since the BaseException converted to an Exception.
There was a problem hiding this comment.
This was copied from CmdLineTask, I think the purpose of this code is to catch non-BaseException exceptions (like raise None) and convert it into standard Exception. I think this is done because multiprocessing fails badly if it sees non-Exception exceptions.
There was a problem hiding this comment.
That might be how Python 2 worked but if I do:
try:
print("About to raise")
raise None
except RuntimeError:
print("Caught RuntimeError")
except Exception:
print("Caught Exception")
except BaseException:
print("Caught BaseException")then I get:
About to raise
Caught Exception
So I think this should be catching BaseException if you want to trap everything including Ctrl-C.
There was a problem hiding this comment.
And I get the same answer with Python 2.
There was a problem hiding this comment.
OK, I did some experimenting, indeed behavior for raise None is (almost) identical between 2 and 3. The difference is for old-style classes used as exception in py2, they are not caught by BaseException or Exception. And you are right that it does not matter for py3, so I should drop that.
andy-slac
force-pushed
the
tickets/DM-13452
branch
from
6189225
to
83f2d5c
Compare
Still WIP, many updates to parser, SuperTask, SuperTaskConfig, unit test
and examples.