Add nextclade tool (nextstrain/nextclade) #101

pvanheus · 2020-09-12T15:15:12Z

nextclade is a command line version of the Nextstrain clade identification and QC service (https://clades.nextstrain.org/).

The tool takes FASTA input and generates a report in JSON / CSV / TSV format. As noted in the TODO I don't know how to pass an output path in (since a File type must already exist as a file on disk), and secondly I don't know how to specify that at least one of the output options must be specified.

This tool pulls from nextclade 0.4.2. I thought it was safer to put a version number in than to leave it out.

I added myself in the dct:creator field because I am the creator of the wrapper (and following Dockstore specification for doing such things). If there is a way to credit the upstream project I would like to do so.

ivan-aksamentov · 2020-09-12T20:32:47Z

Hi Peter @pvanheus, very cool, thanks,

I'd be glad to help to resolve the remaining questions.
How can I make Nextclade more convenient to use? Do you have an example or a set of common patterns/best practices for tool implementers?

I don't know how to pass an output path in (since a File type must already exist as a file on disk)

Could you please clarify the "File type must already exist"? Is this something lacking in out toor or in CWL?

How to ensure that one or more of the following should be provided?

Could we provide (hardcode?) all 3 output flags all the time? Even if user don't need some of the files, there is almost no overhead.

How to allow a path - instead of a simple filename - for output?

Unless there's a bug, Nextclade should accept all kinds of paths: filenames, relative paths, absolute paths. I think even /dev/stdout might work. If not, and if it's useful, I'd be happy to adjust.

If there's something else that would help integration, please let me know, and don't hesitate to @ me any time.

P.S. We just released 0.4.3 with the new --output-tree flag that outputs the tree Auspice JSON v2 (the same three that is displayed in the app)

…xtclade

pvanheus · 2020-09-13T06:41:17Z

@ivan-aksamentov My queries with regards to outputs are, I think, simple misunderstandings of CWL and cwltool (the "reference" CWL executor) on my side. In CWL, outputs are taken from the working directory of the running job - thus the outputBinding and glob. The running job does not have access to arbitrary filesystem paths, so the --output-json from a CWL perspective should be a name, not a path. (cwltool does have an -o flag to determine where output files end up btw)

I have updated by TODO comment accordingly.

ivan-aksamentov · 2020-09-13T20:58:56Z

@pvanheus Okay. I have very little understanding how CWL works, but if there's anything needed to be done on nextclade side for the simpler integration, just let me know.

Could you please also add the new --output-tree flag? This is the 4th type of output I just added to the CLI v0.4.3 yesterday. The format is, so called, "Auspice JSON v2", as explained in "Auspice input" section of https://nextstrain.org/docs/bioinformatics/data-formats
You can classify it as JSON I guess. It's somewhat beefy though, up to a few megs.

Regarding

a way to credit the upstream project

Could we add a link to https://clades.nextstrain.org as credits? This way we could add the actual credits and references right to the app's main page when the time comes.

pvanheus · 2020-09-17T07:02:16Z

Hi @ivan-aksamentov - the latest version of the the PR has the --output-tree flag.

cwl-bot · 2020-10-05T18:32:23Z

This pull request has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/how-to-specify-that-at-least-one-of-a-group-of-parameters-is-necessary/211/1

pvanheus · 2020-10-06T14:01:29Z

I switched to using a record for the group of output options, with each output option itself optional - this forces at least one of the output options to be chosen.

…xtclade

nextclade/nextclade.cwl

mr-c · 2020-12-21T06:20:21Z

nextclade/nextclade.cwl

+      type: record
+      name: output_options_record
+      fields:
+        output_json_filename:


Do all of these filenames need to be specified by the user? Why not take the name root of the FASTA and add the relevant file extension?

The underlying tool has options for different output names for each output file. While I am not sure if that will be used in practice, I have followed the underlying tool in the CWL here.

Nextclade dev here. Let us know, folks, what would be the most convenient set of flags. We are flexible here.
P.S. ping me with @ if I don't reply

For "user" I meant the user of this CWL description. For nextclade itself @ivan-aksamentov you did the correct thing to ask for the filename/path.

Is there a cost to output all these types?

@pvanheus If not then I recommend always requesting all output types via the arguments section building off of the nameroot of the fast afile

Good point: from a workflow point of view, naming of files is not relevant. @ivan-aksamentov if i understand correctly the outputs are typically small and there is no significant cost to asking for all of them?

…ement

nextclade/nextclade.cwl

pvanheus · 2020-12-21T19:39:00Z

As per the style guide for this repo, we prefer to use biocontianers. Do they have a suitable image?

They do not. The Dockerfiles for the tool are in: https://github.com/nextstrain/nextclade/tree/master/packages/cli/docker. If I understand the biocontainers procedure (https://biocontainers-edu.readthedocs.io/en/latest/what_is_biocontainers.html) they accept contributions via pull request but that is up to upstream folks like @ivan-aksamentov.

nextclade/nextclade.cwl

mr-c · 2020-12-22T09:03:38Z

nextclade/nextclade.cwl

+class: CommandLineTool
+
+doc: Assign Nextstrain clades to SARS-CoV-2 sequences and provide QC information
+id: nextclade


Suggested change

id: nextclade

No need for id here

mr-c · 2020-12-22T14:45:35Z

nextclade/tests/nextclade_t1.yml

@@ -0,0 +1,13 @@
+input_fasta:


Suggested change

input_fasta:

sequences:

nextclade/tests/nextclade_t1.yml

ivan-aksamentov · 2020-12-23T14:58:23Z

Never heard of biocontainers. Is that a rebranded Docker or something? I see they require some LABELs. Is that it or what needs to be done there?
Do we contribute somewhere (we probably don't have time for this now) or someone contributes to nextclade (super welcome)?

Is there a cost to output all these types?

No.

if i understand correctly the outputs are typically small and there is no significant cost to asking for all of them?

Correct. Outputs are all computed regardless, and are all small CSV or JSON files. The difference between JSON, CSV and TSV is only format - they contain the same data. Tree JSON might be on a larger side, but that depends on what tree user has provided as input tree.

Future: people asked us to also output aligned sequences, proteins, newick tree etc. This might be added later and might be big.

More future: we rewrite Nextclade in C++

mr-c · 2020-12-28T11:28:13Z

Never heard of biocontainers. Is that a rebranded Docker or something? I see they require some LABELs. Is that it or what needs to be done there?
Do we contribute somewhere (we probably don't have time for this now) or someone contributes to nextclade (super welcome)?

https://biocontainers.pro/

BioContainers is a community maintained docker/singularity container registry+builds built on quay.io, GitHub, (Bio)Conda & Debian.

The quickest route is to get nextclade packaged in the BioConda Conda channel

Add nextclade tool (nextstrain/nextclade)

05543ec

pvanheus added 6 commits September 13, 2020 07:42

Merge branch 'release' into nextclade

c884b60

Update to nextclade 0.4.3, add output formats

95750e0

Merge branch 'nextclade' of github.com:pvanheus/bio-cwl-tools into ne…

a623a37

…xtclade

refer to cwl-runner instead of cwl-exec

15a6a65

Make nextclade tool descriptor executable

79c292f

Update TODO

ce99ba9

pvanheus added 3 commits September 22, 2020 17:08

Updated to 0.5.0 nextclade

cf9a6de

Update to support nextclade 0.7.3

7eada4d

Add inputBinding of output-tree

26e74bc

Group output options in output_options record

33f38d3

pvanheus added 3 commits October 8, 2020 09:04

Merge branch 'release' into nextclade

d9eef77

Bump to nextclade 0.8.1

477f09f

Merge branch 'nextclade' of github.com:pvanheus/bio-cwl-tools into ne…

896993e

…xtclade

mr-c requested changes Dec 21, 2020

View reviewed changes

Bump nextclade to 0.10.0, change to schema.org contributor acknowledg…

fa99bd7

…ement

mr-c reviewed Dec 21, 2020

View reviewed changes

nextclade/nextclade.cwl Outdated Show resolved Hide resolved

mr-c reviewed Dec 21, 2020

View reviewed changes

nextclade/nextclade.cwl Outdated Show resolved Hide resolved

mr-c reviewed Dec 21, 2020

View reviewed changes

nextclade/nextclade.cwl Outdated Show resolved Hide resolved

pvanheus added 2 commits December 21, 2020 18:52

Switch to IANA memory limits for non-biological formats

6e8111d

Remove input_ prefix from inputs

f26ab33

Rename outputs

07f7a25

mr-c reviewed Dec 22, 2020

View reviewed changes

nextclade/nextclade.cwl Outdated Show resolved Hide resolved

mr-c reviewed Dec 22, 2020

View reviewed changes

Add link to upstream as SoftwareRequirement, drop id

746b9a9

mr-c reviewed Dec 22, 2020

View reviewed changes

Align test file with nextclade CWL

ad288a8

Switch to nameroot based output naming

357f758

mr-c approved these changes Dec 28, 2020

View reviewed changes

mr-c merged commit 0e32e03 into common-workflow-library:release Dec 28, 2020

denis-yuen mentioned this pull request May 2, 2022

Improve or replace CWL md5sum repo dockstore/dockstore#4886

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nextclade tool (nextstrain/nextclade) #101

Add nextclade tool (nextstrain/nextclade) #101

pvanheus commented Sep 12, 2020

ivan-aksamentov commented Sep 12, 2020 •

edited

pvanheus commented Sep 13, 2020

ivan-aksamentov commented Sep 13, 2020

pvanheus commented Sep 17, 2020

cwl-bot commented Oct 5, 2020

pvanheus commented Oct 6, 2020

mr-c Dec 21, 2020 •

edited

pvanheus Dec 21, 2020

ivan-aksamentov Dec 21, 2020 •

edited

mr-c Dec 21, 2020

pvanheus Dec 21, 2020

pvanheus commented Dec 21, 2020

mr-c Dec 22, 2020

mr-c Dec 22, 2020

ivan-aksamentov commented Dec 23, 2020 •

edited

mr-c commented Dec 28, 2020

Add nextclade tool (nextstrain/nextclade) #101

Add nextclade tool (nextstrain/nextclade) #101

Conversation

pvanheus commented Sep 12, 2020

ivan-aksamentov commented Sep 12, 2020 • edited

pvanheus commented Sep 13, 2020

ivan-aksamentov commented Sep 13, 2020

pvanheus commented Sep 17, 2020

cwl-bot commented Oct 5, 2020

pvanheus commented Oct 6, 2020

mr-c Dec 21, 2020 • edited

Choose a reason for hiding this comment

pvanheus Dec 21, 2020

Choose a reason for hiding this comment

ivan-aksamentov Dec 21, 2020 • edited

Choose a reason for hiding this comment

mr-c Dec 21, 2020

Choose a reason for hiding this comment

pvanheus Dec 21, 2020

Choose a reason for hiding this comment

pvanheus commented Dec 21, 2020

mr-c Dec 22, 2020

Choose a reason for hiding this comment

mr-c Dec 22, 2020

Choose a reason for hiding this comment

ivan-aksamentov commented Dec 23, 2020 • edited

mr-c commented Dec 28, 2020

ivan-aksamentov commented Sep 12, 2020 •

edited

mr-c Dec 21, 2020 •

edited

ivan-aksamentov Dec 21, 2020 •

edited

ivan-aksamentov commented Dec 23, 2020 •

edited