CommonWL

[gijzelaerr]

Hi! More of a question than a issue. Currently we are modelling our data processing workflow using common workflow language (CommonWL, CWL). We hope this gives us the most portability between the various workflow management tools available. I can't really find any work or talk about integrating pachyderm with CWL. Has there ever been any thoughts about doing this? How likely is it that a 'standard' like CWL would get adopted for pachyderm?

Thanks!

[mr-c]

FYI: http://www.commonwl.org

Chat @ https://gitter.im/common-workflow-language/common-workflow-language

[jdoliner]

CWL is something that's been on our radar for a while and is definitely something we're interested in exploring. A few questions I have:

• you can of course run CWL workflows on Pachyderm by packaging up the runtime into a container and running it like that. Is that appealing as a stopgap measure or does that have too many drawbacks to be useful?
• One obvious drawback is that this wouldn't be distributed. How do CWL implementations distribute the workload? Can they distribute individual steps in the workflow or just the different nodes of the workflow?
• Would running a side cluster of one of the implementations on their site be an appealing way to run CWL workloads? This is a patterns that's emerging in Pachyderm right now for incorporating other protocols into pachyderm pipelines.

[mr-c]

Cool, glad to hear it.

1. FYI: CWL is a standard, not a runtime. True, one could use any CWL implementation inside a container treating the entire workflow as a single step, but then you'd lose granularity.
2. CWL implementations distribute their workload according to implementation specific heuristics as they may be running on a single node, high throughput computing cluster (grid), or one or more cloud providers. The CWL model is rich enough to provide them with lots of information to make these choices. Some implementations will implement CWL's scatter/gather across nodes, yes.
3. I'll leave this for @gijzelaerr and others to answer, but from what I can see there should be enough information in CWL descriptions to convert into Pachyderm workflows.

Let me know if you'd like a real time video chat, I'd be happy to go into more depth and share about how various projects have implemented CWL support so far.

[samuell]

Just to chime in with my 5c: Based on my experimentation with CWL and my breif reading of Pachyderm workflow examples, I get the impression that they are actually very similar.

I would even think just a converter from CWL yaml/json to Pachyderm yaml, would not be very hard at all.

CWL is also doing the definition of inputs / outputs (in-ports / out-ports in some contexts) in a more re-usable and composable way, as far as I can see, so I think pachy could probably learn a thing or two there as well, for any future updates of its format.

But all in all, the formats seem very similar.

[OliverEvans96]

Any further thought here? Has anyone attempted a CWL to Pachyderm yaml converter? Does it seem like there's sufficient feature parity for this to work well? I think it would make Pachyderm quite appealing to interface smoothly with the CWL standard.

[dannykwells]

Big Pachdyerm user here - we are using it throughout all of our pipelines. Are there any plans to support CWL soon? This language is becoming the lingua franca of computational genomics and support for it is extremely important for our continued use of Pachyderm.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gijzelaerr]

well that would be a shame :)

[marcadella]

As stated above it'd just be a matter of implementing a converter from CWL to pachyderm's pipeline def. I think it would be quite useful for pachyderm's community indeed.
A quick google search did not bring up any results. Has anyone seen a project somewhere?

[mr-c]

@marcadella I'm not aware of any, but that doesn't mean that such a thing hasn't been started.

I am happy to assist anybody writing a CWL converter!

[JoeyZwicker]

While I'm not very familiar with CWL, I'm not actually sure if this would be very easy to implement as there are many concepts that don't seem to translate cleanly from CWL to Pachyderm.

A first implementation of this would make for an excellent PR from someone who's more familiar with CWL and we'd be happy to give further guidance from there.