Pipes should fail after an exception

[kstrauser]

In my opinion, pipelines should fail when a Job's function raises an exception. For example, I imagine a Pipe with [Job(build_a_thing), Job(test_a_thing), Job(deploy_a_thing)]. If the test fails, we should never get to the deploy step. That's not now it works now, though:

from pypette import Pipe, Job

def func1():
    1/0

def func2():
    print('in func2')

pipe = Pipe('Simple pipe')
pipe.add_jobs([
    Job(func1),
    Job(func2),
])

pipe.run()

In this case, func2 will be executed even though func1 explodes. If the two run in parallel, then func2 should be allowed to finish (that is, don't say "oh, func1 died! time to kill all the other Jobs!"). But given this scenario:

def func3():
    print('in func3')

second_pipe = Pipe('Simple pipe')
second_pipe.add_jobs([
    Job(func1),
    Job(func2),
], run_in_parallel=True)
second_pipe.add_jobs([Job(func3)])

pipe.run()

I don't think func3 should ever be called because one of the Jobs in the first step of the pipeline failed.

[csurfer]

@kstrauser: I have a different view to present here. I don't think it should fail. The reason I think so is one of the jobs might be a cleanup job which is quite needed in a lot of places. If the failure of a stage compels the next stage to fail, we may end up in a situation where the cleanup jobs are not executed.

Having said that I believe we should have control over dependency and choice. So, I have been thinking of Job having an optional parameter called gate. This gate is a callable of some sort which returns a boolean which would decide whether the job needs to be executed or not.

j1 = Job(job1)
# Lets assume job1 pushes its state into a queue or some variable x.
def gate_method():
    flag = True
    # flag value is set to variable x or popped value from queue
    return flag
j2 = Job(job2, gate=gate_method)

When job 2 is being run it will check for value returned by gate_method and only if it is true will execute job2.

That will introduce the notion of dynamic decision in pipeline. What do you think?

[kstrauser]

That’s a really good point about cleanup jobs! I think the gate idea sounds wonderful, though. What if there was also a function like last_job_succeeded() that returns true if and only if the previous job completed without raising an exception. Perhaps that could even be the default value, like gate=last_job_succeeded, so that by default a failure stop execution but that it would be trivially easy to turn it off for cleanup jobs?

[csurfer]

@kstrauser : Have added a bunch of new stuff including exception handling modes and dependency. Look at examples/ for the usage. Still testing a couple of things after which will release officially. Give it a whirl and let me know.

[csurfer]

v0.0.9 should resolve this issue.

[kstrauser]

I'll take a look. Thanks for following up!