While it's a more thorny proposal for things like parallel execution (but likely still do-able), normal serial execution in execute() should be storing the results of each host+task execution run, and returning e.g. a map of host string to result value, instead of None.
This way folks that do rely on tasks returning real values are still able to obtain that data without stuffing it into env.
Submitted on the mailing list by K. Danilov.
Any chance the serial execution case might make its way into 1.4?
I'll add it for now because a few folks have asked about it and it should be quick to at least implement the serial aspect. But I might bump it back to a later release if the other dev for 1.4 takes too long, be warned :)
Initial work on execute() return values (#474)
Taking a stab at this now (sorry @markatto, yours is already a bit out of date due to other 1.4 changes :(). Remaining todos:
Preserve retval of execute()d tasks in base/serial cases
More tests re #474
Use exception as return value for skipped-over network errors
First draft of parallel+queue for return values
Changelog/docs re #474
This has been merged into master and should be all set, including parallel execution case. @markatto @cloax if you guys want to nab latest master as of now and give the feature a shot, that'd be great!
Awesome! Will give it a look. Much thanks!
In addition to the abovementioned issue, I found another one: large flights can now cause OSError 24, Too many open files, apparently due to the new Queue objects being used. (Commenting out just the use of the Queues, makes the error go away.)
Too many open files
Need to see if it's possible to only create the queue at runtime instead of prep time; this would make JobQueue less Fabric-agnostic but honestly that's a pipe dream and it's already caused some wonkyish code, so at this point I think I will just cross the streams. We'll see.
I think I have fixed the OSError but found another semi related issue. In fixing the OSError I noticed that subprocesses could raise Exceptions which then A) print all ugly and such, and B) result in no return value whatsoever from the subprocess.
To match how we handle serially executed tasks (which will return the return value or the exception preventing return) I updated things to capture + store any exceptions. However, I then appear to run into this pickle/multiprocessing bug which creates TypeErrors when pickle somehow tries to re-initialize the exception with the wrong arguments.
Will keep poking to see exactly what it is doing and if there is any way to handle it on our end. To safely handle arbitrary exceptions without passing the exception itself would be difficult at best -- maybe just return the class name or something, which would at least be better than nothing.
Move to single global queue to avoid OSErrors.
Deal with silly pickling behavior of captured exceptions
Things should be cleared up now, though I'm not 100% sure about #547 (but it looked strongly related to tonight's changes.) That'll be handled there; this is closing again.
Don't hide subprocess exceptions when returning them. Jeez!
Just wanted to say thanks again for knocking this one out. Has been working great for my needs!
@cloax Glad it's working for you! :) always great to get positive feedback.