Decouple node and edge creation

[CJ-Wright]

Would it be possible/desirable to separate node and edge creation?

When we create a streamz pipeline we are creating a directed task graph. Currently this is done by creating nodes and simultaneously attaching them to other nodes (creating an edge). However, we may run into the situation where we want a bunch of nodes sitting in a library waiting to be used. This way users could mix and match how their nodes connected, essentially adding edges after the nodes were imported.

[mrocklin]

Can you motivate the reasons behind creating nodes before attaching them? Why not create at the time of attachment? I don't think I have an intuition for your use case.

…

On Thu, Aug 17, 2017 at 5:40 PM, Christopher J. Wright < ***@***.***> wrote: Would it be possible/desirable to separate node and edge creation? When we create a streamz pipeline we are creating a directed task graph. Currently this is done by creating nodes and simultaneously attaching them to other nodes (creating an edge). However, we may run into the situation where we want a bunch of nodes sitting in a library waiting to be used. This way users could mix and match how their nodes connected, essentially adding edges after the nodes were imported. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#56>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszKkmtAjb1NCUMCYlodIpgPb_JAufks5sZLNIgaJpZM4O62Np> .

[CJ-Wright]

I create a few nodes which map some function onto the data. Then I'd like to write 5 pipelines which use these nodes in different positions in the execution. Currently I'd need to duplicate the code for those nodes 5 times, since their upstream nodes would be different and that is bound at instantiation. With nodes created without a graph I could just import the nodes into the 5 pipelines and just plumb them together.

[mrocklin]

But the nodes would be the same objects, no? Wouldn't they inadvertently feed data between the pipelines?

…

On Thu, Aug 17, 2017 at 7:51 PM, Christopher J. Wright < ***@***.***> wrote: I create a few nodes which map some function onto the data. Then I'd like to write 5 pipelines which use these nodes in different positions in the execution. Currently I'd need to duplicate the code for those nodes 5 times, since their upstream nodes would be different and that is bound at instantiation. With nodes created without a graph I could just import the nodes into the 5 pipelines and just plumb them together. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#56 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszNcJeESy1g9FsYdw-c5dqU-gA1wJks5sZNIGgaJpZM4O62Np> .

[CJ-Wright]

Maybe. That would only happen if I was running all the pipelines at once. One could run a deep copy on the node to make copies?

[mrocklin]

Is it costly to make a new node each time as is currently done? What are the advantages of pre-instantiating these objects ahead of time rather than creating them on the fly? So far in my toy examples I've only run into cases where recreating every time was free.

[jrmlhermitte]

how does this differ from connect?
(#45)

[CJ-Wright]

Connect may have solved this issue, we just need to have a unique source for every incoming node to connect to.