How to use standard to update a workflow?

[vsoch]

Hi! We're looking to use the spec hear for a monitor for Snakemake, and I'm wondering if the API supports any kind of update_workflow endpoint, and if not, is there a reason? Thank you!

[vsoch]

And for authentication, I'm used to a server returning 401 with a Www-Authenticate header to request authentication. Is that something supported here? I saw the auth_instructions_url but it wasn't clear if that is for a human readable URL or should have the content of Www-Authenticate.

[patmagee]

hey @vsoch WES as a specification is relatively unauthoritative around the specifics of how an implementation does things like authentication or even if they DO authentication. There is no "ONE" implementation of WES, instead, you as an implement, will construct your API in a way which conforms to the standard.

When it comes to the update_workflow endpoint, no such one exists. CWL, WDL and Nextflow dont have concepts of updating a running workflow. When you submit a run it represents a discrete execution that really should not be modifiable by the user, they just run to completion (or failure). If you would like to update a workflow, you change the descriptor files that were submitted with the run, and the resubmit a new run

[vsoch]

I don’t mean update like that, I mean the running workflow unfair the API with an updated status. Is that possible?

[patmagee]

If I follow you correctly, you are hoping to use WES then to track a Snakemake workflow and update the state that the WES api reports?

So, the WES Spec is not an implementation but a specification. This means that there is no functional code associated with the spec and it is not something that you can just spin up and it will do things for you. Instead, implementors are responsible for either adapting existing services or creating a new one to fit their needs while conforming to the WES specification.

To that end, You will need to create an application which exposes a WES API and can full fill your needs (its worth pointing out that I do not think snake make is an officially supported language for WES). How specifically it tracks a snakemake execution, launches a execution etc, is completely implementation specific and you have free reign to figure this out. Other implementations have internal Databases and service which orchestrate execution and manage state.

[ruchim]

@vsoch -- happy to have a call to help verify anything. If you're looking for enhancements to Snakemake code or another library codebase you're utilizing -- I'd be happy to try.

[vsoch]

@patmagee sorry I didn't respond to your comment! To be clear, I'm asking about why there is a hole in the spec with respect to updating, and I'm in full understanding that the spec here does not include code, and it's up to users to do the implementation. My question is around why the spec doesn't have support (or definition) for any way to update a workflow. It seems that the expected interactions are to create and then get, but not update. If we need an update endpoint, it would be part of our custom implementation and not following the spec. My desire would be to add this sort of update endpoint to the spec, as it seems important. Have other users not needed any sort of update operation?

[patmagee]

@vsoch its important to remember that submitting a run through the POST /ga4gh/wes/v1/runs endpoint is not the same as creating a workflow. Instead it is executing an instance of a workflow definition which is defined elsewhere. IN all implementations that I have used, (using both WDL and CWL), once a workflow execution has started, there is not way for them to be updated. A workflow execution does not represent the workflow itself

I think what you might be thinking of is more along the lines of the Tool Registry Schema. This service allows you to create a definition of a workflow, update it, create different versions of it, and provide a referencable URL to use elsewhere. IS this more along the lines of what you are thinking of?

[vsoch]

Oh that's interesting - let me ping my colleague that works on Panoptes (and we were originally looking for this endpoint) - @fgypas what do you think?

[uniqueg]

Never saw this issue, but I know Panoptes d @fgypas. I think there's still a bit of a misunderstanding here, and while perhaps it may perhaps not matter to @vsoch anymore at this point, I thought I would try to clear it up.

WES doesn't have an update status endpoint because a typical client shouldn't have the ability to update the workflow status - it's the job of the workflow engine executing the workflow, and that's happening on the server side of WES. The WES spec does define workflow execution states and return them via the logs and a dedicated endpoint for clients to consume. These states imply that WES implementers update workflow execution states, and typically that's what implementers do (including the Snakemake WES implementations that I know, which by now became available), although they MAY choose not to do so. If they internally want to use an endpoint to update states, or directly update the states in the database is an implementation detail that the WES specification does not want to interfere with.

From a Panoptes point of view, where (I suppose) you want to track Snakemake execution via a dashboard, I think the GET /runs/{id}/status and GET /runs/{id} endpoints should be all you need. Well, and a Snakemake WES implementation that updates states, like WESkit, for example.