Passing output from last task into new task #5
Comments
@mafintosh @maxogden your thoughts? making Also, forcing it to be a stream would help with async - #4. Or just pass in params lol.. |
|
did not actually do this, was resolving from fs |
@mafintosh I attached a function I tried attaching a bare object first (see closing commit before reopen), from inside task, after end/finish, but then the end/finish handler from join doesnt get it not sure if theres another way to do it when you can just attach an object rather than a closured function |
For duplex streams, this is obvious.
But for tasks that create files:
We wait for the task to finish:
inside Task
Then it is
Join
's job to pass these on:But nothing is being passed in yet.
Idea 1
Make
Task
always be a duplex object stream. It's first and final chunks will be objects pertaining toinput
andoutput
. Then we need to wrap the actual stream of the task, for example, a Readable whosethis._source
is just another stream. This may complicated doing duplex streams piping together.Idea 2
Wow. This feels so obvious now. Pass items in the first param of
Task
. (which right now is just() => { ... }
. I'll say my third idea anyways lol.Idea 3
Have a
task.made
event emitted that emits a callback which can then pass in resolved outputs which become resolved inputs in the new task.It is important to handle this case well, because with these pipelines, not everything is streamable. For example, doing an alignment requires an indexing on the reference which needs to run fully. But I think having Task always be a stream (i.e. it wraps promises into streams) makes it easier to then orchestrate things. And having an output as a
File
tells the orchestrator that this Task cannot be piped into another - unless we go the route where Task is always a duplex stream that wraps over other streams.The text was updated successfully, but these errors were encountered: