workflow_as_node does not update the input (if it was run already)

[djarecka]

If I create a wfnd first and later use within wf that has different input it works properly only if wfnd was not run before wf. See the tests:

@pytest.mark.parametrize("plugin", Plugins)
def test_wfasnd_wfndupdate(plugin):
    """ workflow as a node
        workflow-node with one task and no splitter
        wfasnode input is updated to use the main workflow input
    """

    wfnd = Workflow(name="wfnd", input_spec=["x"], x=2)
    wfnd.add(add2(name="add2", x=wfnd.lzin.x))
    wfnd.set_output([("out", wfnd.add2.lzout.out)])

    wf = Workflow(name="wf", input_spec=["x"], x=3)
    wfnd.inputs.x = wf.lzin.x
    wf.add(wfnd)
    wf.set_output([("out", wf.wfnd.lzout.out)])
    wf.plugin = plugin

    with Submitter(plugin=plugin) as sub:
        sub(wf)

    results = wf.result()
    assert results.output.out == 5
    assert wf.output_dir.exists()


@pytest.mark.xfail(reason="wfnd is not updating input for it's nodes")
@pytest.mark.parametrize("plugin", Plugins)
def test_wfasnd_wfndupdate_rerun(plugin):
    """ workflow as a node
        workflow-node with one task and no splitter
        wfasnode is run first and later is
        updated to use the main workflow input
    """

    wfnd = Workflow(name="wfnd", input_spec=["x"], x=2)
    wfnd.add(add2(name="add2", x=wfnd.lzin.x))
    wfnd.set_output([("out", wfnd.add2.lzout.out)])
    with Submitter(plugin=plugin) as sub:
        sub(wfnd)

     wf = Workflow(name="wf", input_spec=["x"], x=3)
    # trying to set before
    wfnd.inputs.x = wf.lzin.x
    wf.add(wfnd)
    # trying to set after add...
    wf.wfnd.inputs.x = wf.lzin.x
    wf.set_output([("out", wf.wfnd.lzout.out)])
    wf.plugin = plugin

    results = wf.result()
    assert results.output.out == 5
    assert wf.output_dir.exists()

Haven't debug too much, but it looks like in the second test the input wf.wfnd.inputs.x is not taken from the wf.inputs (it shows x=LF('wf', 'x')), so the wf.wfnd.add2 takes the previously set input, i.e. 2

[satra]

@djarecka - isn't this the same thing we have discussed, which is that the graph of the workflow changes on run ? so the lz pieces are no longer available?

[djarecka]

@satra - it is probably related, but at that time i fixed all the test that i came up with. Didn't realize that situation changes when the inner wf was already executed... See, the first test, this one is running fine

…

On Tue, Jul 30, 2019, 19:31 Satrajit Ghosh ***@***.***> wrote: @djarecka <https://github.com/djarecka> - isn't this the same thing we have discussed, which is that the graph of the workflow changes on run ? so the lz pieces are no longer available? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#107?email_source=notifications&email_token=AAMV6GUZSHS6GOPI7JAIPATQCDFORA5CNFSM4IH7K6PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3FTPPI#issuecomment-516634557>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMV6GUE6RB4QDXNBCXDWHLQCDFORANCNFSM4IH7K6PA> .

[satra]

as a test you could:

1. create a copy of the graph of wfnd (before running it)
2. run the workflow
3. reset the graph to the copy

and see if this crashes the second one

[djarecka]

@satra - i believe that's what you ment:

...
    wfnd_graph = deepcopy(wfnd.graph)

    with Submitter(plugin=plugin) as sub:
        sub(wfnd)

    wfnd.graph = wfnd_graph
...

but it doesn't change anything.

[djarecka]

@satra - I've been thinking about it and for now I'm not completely sure what is the best way of changing this. Regarding your question (yesterday, when I was answering, I thought that you were referring to a different discussion), yes the graph does change, but I really believe that how we should do. Maybe we could have a way to "reset" the nodes when they are added to a next workflow, but maybe for now we can just check in the add method if the node has already result and raise an exception?

[satra]

@djarecka - this is not going to be an uncommon situation, and we need to understand what needs to be adjusted for this to work. i'm glad you put that test in. even though we can raise an error for now, this should technically work.

[djarecka]

I believe we have to add a mechanism to reset inputs and remove results for any task (and all inner task if this is a workflow) if any changes to the task is introduced. I've added some simple changes, but didn't propagate inside the workflow, so have to spend more time on this.

But if you have any other suggestion, let me know, I'm not fully convinced. However, I'm still against keeping the graph unchanged during the execution (from my experience this introduces more problems than it solves, but maybe I was doing something wrong...)