adding microbiome analysis workflows to IWC with test data

[EngyNasr]

I tried to reduce the test-data in this PR, hope it works.

Thanks a lot,
Engy <3

[mvdbeek]

Thanks, can you add a README, Changelog and dockstore.yml files ? (https://github.com/galaxyproject/iwc/blob/main/workflows/README.md#structure-of-the-directory)

[EngyNasr]

@mvdbeek did I miss something else ?

Thanks a lot for helping me out :)

[EngyNasr]

@wm75 Can you help me revising and merging this PR
Thanks a lot <3

[EngyNasr]

To do as discussed with @wm75 :

1- remove "latest" from workflow names
2- keep only one file to test for all the workflows, except the pre-processing
3- pre-processing tests we can use the txt file to compare content
4- make a readme inside each folder as well
5- make the changelog only inside each folder
6- remove the big docker.yml in the main folder and keep only the one in every folder

[EngyNasr]

@bebatut I have added the 5 workflows for the single samples run and the 4 workflows for the collection of samples run so in total 9 workflows with their test data, I have chosen the minimum size sample data which contain VFs, Contigs, etc. the maximum file size is 50Mb, but the other files are either in Bytes or Kbs

[wm75]

@EngyNasr @bebatut I don't see much value in offering the single-sample workflows, when collection-based flavors exist that could be run with 1-element collections. Having the single-sample WFs published would just mean more maintenance and synchronization efforts.
Or is there anything that can be achieved with the single-sample versions, that the collection-based versions don't do?

[wm75]

@EngyNasr can you please run some json reformatter tool over your workflows. Single-line JSON is just not very nice to review and prevents meaningful diffs. Use, e.g., python3 -m json.tool collection-version/pathogen-detection-nanopore-pre-processing-collection/Pathogen-Detection-Nanopore-Pre-Processing-collection.ga collection-version/pathogen-detection-nanopore-pre-processing-collection/Pathogen-Detection-Nanopore-Pre-Processing-collection_pretty.ga or any other tool you like.

[wm75]

Some initial comments:

• At least some of your workflows are lacking a release attribute, which you need to add manually.
• In the Preprocessing workflow, the conversion of fastq.gz to plain fastq, just for the purpose of filtering reads by their ID, is rather unfriendly for a user's quota.
  There is toolshed.g2.bx.psu.edu/repos/iuc/seqtk/seqtk_subseq/1.3.1 which can filter compressed fastqs directly (and which is probably faster in all cases). It's downside is that it only keeps matching IDs, but can't discard them, or write both to separate files (like toolshed.g2.bx.psu.edu/repos/peterjc/seq_filter_by_id/seq_filter_by_id/0.2.7).
  So if you want the non-host reads, you'll have to invert the action of Filter Tabular at the step before.
  If you really need also the host reads as a separate file (which I'm not entirely convinced of), you would have to run Filter Tabular and seqtk subseq twice, but even that might still be better than the current way?

[EngyNasr]

@EngyNasr @bebatut I don't see much value in offering the single-sample workflows, when collection-based flavors exist that could be run with 1-element collections. Having the single-sample WFs published would just mean more maintenance and synchronization efforts. Or is there anything that can be achieved with the single-sample versions, that the collection-based versions don't do?

it was just the old way we used to do the analysis and we use these workflows in the current training material, thats why we wanted to have both as two versions of the workflow. but definitely they are useless now since the collection version does the same exact job, but it will never take a single file it has to always be a collection

[paulzierep]

This tool should also work here: toolshed.g2.bx.psu.edu/repos/iuc/krakentools_extract_kraken_reads/krakentools_extract_kraken_reads/1.2+galaxy0
Can use fastq.gz

[EngyNasr]

I need help in tests @wm75 @paulzierep @bebatut :

• In the Pathogen-Detection-Nanopore-All-Samples-Analysis.ga workflow, I use __FILTER_EMPTY_DATASETS__ (version 1.0.0) which seems not installed when I run planemo test.
• Same for Pathogen-Detection-Nanopore-Gene-based-pathogenic-Identification-collection.ga, workflow for Build list tool

I noticed that the tool id for these tools is not like the rest of the tools e.g. toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_getfastabed/2.30.0+galaxy1

Is there a way to solve that for these tests to succeed?

[mvdbeek]

@EngyNasr once galaxyproject/planemo#1377 is merged and a new version is released it should work.

[EngyNasr]

@EngyNasr once galaxyproject/planemo#1377 is merged and a new version is released it should work.

thank you so much :)

[EngyNasr]

@EngyNasr once galaxyproject/planemo#1377 is merged and a new version is released it should work.

@mvdbeek, Is it possible that it is also done for FILTER_EMPTY_DATASETS ?

[bebatut]

I think it was closed by mistake

[mvdbeek]

Looks like it worked and you only need to work on your test assertions.

[wm75]

@EngyNasr two questions on the latest preprocessing version:

• why is the host ID now not a WF parameter anymore? You could use the "Map parameter value" expression tool to map a user-selected species name to the required taxid.
• is there a specific reason why you're using fastqc, when you're already running fastp? MultiQC can also visualize fastp JSON output dataset and it doesn't look too different from what two FastQC results (before and after trimming).

[wm75]

The Pathogen-Detection-Nanopore-All-Samples-Analysis WF needs much better annotations and input dataset labels to make it understandable what it is good for. Right now, understanding the purpose without knowing the tutorial is really hard. Even the README file only talks about VF genes, but many people wouldn't know what that is.

[mvdbeek]

The one error you've got is still the same:

Loading database information...Failed attempt to allocate 65426688000bytes;
you may not have enough free memory to load this database.
If your computer has enough RAM, perhaps reducing memory usage from
other programs could help you load this database?
classify: unable to allocate hash table memory

you'd need a smaller database for testing, I think @wm75 or @paulzierep were interested in doing that ?

[EngyNasr]

The one error you've got is still the same:

Loading database information...Failed attempt to allocate 65426688000bytes;
you may not have enough free memory to load this database.
If your computer has enough RAM, perhaps reducing memory usage from
other programs could help you load this database?
classify: unable to allocate hash table memory

you'd need a smaller database for testing, I think @wm75 or @paulzierep were interested in doing that ?

I can use the Standard-8 or Standard-16 databases instead for now, I just some administrative help to add to the kraken2 tool on Galaxy EU and Org.

@mvdbeek
Another question, the HTML output of the Github testing, shows an error in the Genebased Pathogen identification workflow too: Expecting value: line 1 column 1 (char 0), which actually blocks the runs of the Pre-processing and the taxonomy profiling workflows.

this error I dont understand, can you please check that, or is it only the database size problem of the taxonomy profiling workflow?

[EngyNasr]

Dears :) @wm75

This PR is finally ready for review, it would be great if we could merge it this week

Thanks a lot,
Engy

[bebatut]

Thanks @EngyNasr
I made some comments mostly for the 1st workflow, but they apply also to other workflows

[EngyNasr]

Thanks @EngyNasr I made some comments mostly for the 1st workflow, but they apply also to other workflows

Thank you so much I will follow them all and ping you again, thanks a lot