-
-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: Cancellable task pipelines #142
Comments
This looks really good. It confused me a lot initially, but it makes sense now (I think). Would I be correct in saying that the main benefits (over a single hard-coded task) are:
So another example would be something like Hipmunk's search page, where upon kicking off a search, there's a dynamic list of airlines that get searched. And that dynamic list of search tasks could be represented as a pipeline? |
Yup, those 2 points are exactly it! We could think of a pipeline as a higher order task. In addition, I think regular application and UI logic is also great fit for a pipeline: For example:
They're not necessarily data pipelines by any means, but a series of discrete events that must happen in a specific order. |
@poteto this RFC is super well written. Well done 👏 . This looks super promising. My single concern is how you would properly handle failures in a pipeline step. It looks as though you have to call
I think we can maybe extend the requestGeolocation: step(task(function* () {
let coords = yield getCurrentPosition(); // simple example
return coords || cancel();
})).onCancel('Could not find your location') This sort of API can guarantee an error message across multiple failure states. |
@offirgolan I would imagine that |
Yes I forgot to add to the RFC, in ember-pipeline it also accepts a function. The cancel token can receive a value of any type as well, and isn't limited to a string. |
I'd prefer a fast error, maybe a warning
Technically, it's possible for us EC's task runner to treat a returned or yielded
Seems like this is something that would/should exist for all TaskInstances in general? Can you provide an example of how
For better or for worse the EC codebase has standardized on single getStores confusion
Why does the function you pass to Also, I wonder if Using pipelines to prevent potential collisionsOne nice thing about pipelines that might solve an issue I've encountered: Say you have two buttons that each fire two separate pipelines that each perform a task on a component before hitting the same task on a Service, which changes some app state; likely, you'll want to prevent one task from being startable of the other is already "on its way" to changing app state. Since pipelines make you declare up front all the "steps" you wish to take, we could potentially use this information to expose an API to make pipeline B unperformable while pipeline A is still running. The underlying mechanism here, really, is essentially grabbing a lock on each of the tasks you might end up calling. It's more derived state that can be used to coordinate locks on state changes. General ThoughtsLove the derived state. Definitely agree that regardless of whether we do pipelines or some kind of async FSM, there should be conventions for following a happy path / progression of steps. I still kinda feel like this should be implemented on top of an fsm primitive? I'm thinking out of the Rails e-commerce platform Spree does it: they use the state_machine gem for stashing the current Order state, but then they let you define a checkout_flow that overlays the states, optionally omitting certain steps in the "pipeline" depending on store configuration. |
Thanks for the feedback!
👍
Returning makes the most sense to me, yielding seems semantically incorrect for cancelling a generator.
I'm not American, but I can live with that I guess...
My assumption was that because a pipeline is also a task it would have a similar API (i.e. it expects a generator fn. I suppose it doesn't necessarily need to be a generator, but I wanted to highlight that it would have concurrency policies as well (which currently lives on the
It's probably a topic for another RFC tbqh. Also re: other derived state, definitely! I think there is much more state we can derive from a pipeline, I'm sure more will emerge after we have some kind of prototype in people's hands. I think an FSM primitive could handle creating a pipeline, but would it be overkill? I have more thoughts on this, I'll add them in a separate comment |
@poteto have you had any more thoughts on this? It might help to add a full example of a pipeline to help visualize how arguments / and return values flow through the pipeline. What does it look like to convert a task that used to |
@machty I just updated the RFC with a more "real-world" example (see The idea is that tasks shouldn't know (or call other tasks) when part of a pipeline. So each task should only return the values (the "subject") that makes sense for that task - for example, a validation task would return These additional arguments should be passed when constructing the pipeline via |
Summary
Cancellable pipelines are a higher order Task with new derived state and the ability to cancel intermediate tasks while the pipeline is being executed. I have implemented a proof-of-concept that handles regular functions in
ember-pipeline
.Motivation
When using tasks in an application, it is a common use case to have a "happy path" series of functions that require some prerequisite before another function can be called. For example:
In the above, it doesn't make sense for the other tasks in
getStores
to fire if the user has not granted permissions to share their location – hence the need for checking if coordinates were received by the taskrequestGeolocation
.However, this approach is error prone and leads to an abundance or overuse of conditional logic and other defensive programming techniques that makes tasks and functions harder to debug.
I propose a new kind of
ember-concurrency
primitive that represents a serial collection of tasks – thepipeline
. Apipeline
is similar to a task group in the sense that it it also a kind of (higher order) task that has task state. If at any point a task yields a cancellable token, the entirepipeline
cancels and derived state for thepipeline
reflects that.This approach means that users will no longer need to program so defensively in order to perform a sequence of tasks that depend on the result of the previous task.
Pipelines are inspired by railway oriented programming and can make complex task chains easier to reason about, and they also provide an opportunity to handle cancellations in a clean way.
Detailed design
Here's what I'm proposing the API will be for a
pipeline
with aComputedProperty
prototype, along with helper functionsstep
andcancel
:Step
A
step
is a wrapper around a task that can hold additional metadata and runtime configuration for that task. For example, this could allow for passing extra arguments:When a task is included in the pipeline as a
step
its concurrency policy is ignored in favor of the pipeline's.Cancellation
In the above, the first step in the pipeline
requestGeolocation
can yield/return a specialcancel
token with an optional reason, if for example the user has declined to share their location.Once the pipeline receives that token, it aborts the rest of the chain, and passess off the cancellation object to an optional cancel handler. Here, the user will be able to match against the task's name (or other property, for example the cancellation reason) and then handle it explicitly.
Other yieldables
It would also be possible to yield a
pause
token that returns a function/task that can be used to resume the pipeline at a later point.Derived state
A pipeline will also have new derived state. For example:
Composition
Finally, a non-
ComputedProperty
pipeline
could also be composed at runtime. For example, consider the case in a highly dynamic form wizard:And of course, it would not be real composition if you could not also compose pipelines (but this may / may not make implementation very complex):
More complete/real-world example
How we teach this
Documentation and guides will have to be updated accordingly. I believe that
pipeline
s will be straightforward to teach given enough examples on when to use one. TheComputedProperty
form of thepipeline
is also very similar in API to a TaskGroup.Drawbacks
It may not be immediately obvious why one would use a pipeline over a regular task. It will be up to documentation and good examples to showcase their benefits.
Alternatives
As discussed in Slack, a state machine primitive could be more useful and also satisify the same motivations.
Unresolved questions
Should this be a part of
ember-concurrency
, or could it live standalone as a companion addon?I may have gotten some of the API wrong, but I think the "spirit" of the implementation should be clear. Please feel free to correct if I have made a mistake!
The text was updated successfully, but these errors were encountered: