Add rgcca #3179

ecamenen · 2020-09-02T15:37:12Z

FOR CONTRIBUTOR:

- I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
- License permits unrestricted use (educational + commercial)
- This PR adds a new tool or tool collection
- This PR updates an existing tool or tool collection
- This PR does something else (explain below)

…al visibility

bernt-matthias · 2020-09-17T09:19:28Z

@ecamenen Can you fix the R linting and tool failure issues?

ecamenen · 2020-09-29T16:20:07Z

@bernt-matthias it should work now

bgruening

Thanks @ecamenen! Small review inline!

bgruening · 2020-09-29T19:56:17Z

tools/rgcca/TUTORIAL.md

@@ -0,0 +1,236 @@
+# TUTORIAL RGCCA GALAXY-TOOL 
+


Should this be moved to the GTN? training.galaxyproject.org/

That would be the goal. After validation of the PR? I'm ending my contract soon without being sure that I'll be renewed and I don't know how much time I could still give for the new adventure which is creating a Galaxy tutorial. is it possible to keep that here at the moment ? if not, I can make a link in the help section to the tutorial of my git company .

Can you dump this file as PR against training.galaxyproject.org ... just as a WIP PR. And hopefully, you will get paid to work on it in the future. In this repo it will be overseen :(

Refresh problem, I hadn't seen the comment. OK, no worries, I will remove it and send it to a tutorial link of my institute.

bgruening · 2020-09-29T19:57:53Z

tools/rgcca/rgcca.xml

+                help="A block defined as the concatenation of all the other blocks. The space spanned by global components is viewed as a compromise space that integrated all the modalities and facilitates the visualization of the results and their interpretation. If disabled, all blocks are assumed to be connected or a connection file could be used."/>
+
+            <conditional name="supervised">
+                <param name="bool" type="boolean" label="Use a block as response (supervised analysis)"/>


Can you use a select here instead of a boolean? See https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#booleans

bgruening · 2020-09-29T19:59:13Z

tools/rgcca/rgcca.xml

+            <conditional name="supervised">
+                <param name="bool" type="boolean" label="Use a block as response (supervised analysis)"/>
+                <when value="true">
+                    <param name="block_response" type="integer" value="0" min="0" max="10" label=" " help="@BLOCK_RULES@"/>


But if you only have one parameter, you can remove the entire conditional I think and make the integer just optional="true"

I am not very keen about that. For integer, the user cannot go back if he has selected an integer for this parameter and then realizes that finally he doesn't want to select any. He has to refresh the analysis for that. If possible, I would prefer to stay on a conditional select option.

Oh, if this is the case, there is a bug in Galaxy frontend. ping @guerler is that something you could look at?

Can't you just clear the text input field?

bgruening · 2020-09-29T20:00:18Z

tools/rgcca/rgcca.xml

+        <data name="design"  label="design.pdf" format="pdf"/>
+        <data name="individual_table"  label="individuals.tsv" format="tsv"/>
+        <data name="variable_table"  label="variables.tsv" format="tsv"/>
+        <data name="R_results"  label="rgcca.result.RData" format="rdata"/>


I would write all variable names in small letters

bgruening · 2020-09-29T20:00:56Z

tools/rgcca/rgcca.xml

+            <!--</output>-->
+        </test>
+
+        <test expect_num_outputs="8" expect_exit_code="0">


Super cool! Thanks for the great tests.

bgruening · 2020-09-29T20:03:50Z

tools/rgcca/rgcca.xml

+                <param name="bool" type="boolean"
+                    label="Load the design matrix (if superblock or supervised disabled)"/>
+                <when value="true">
+                    <param name="file" type="data" format="tsv,tabular,txt,csv" label=" "


tsv is not a csv? Does this matter? In genearl I would recommend to use only TSV in Galaxy. There are so many tools that can work with TSV.

the default format when I upload my files to planemo is tabular. Do I leave only tsv as the accepted format for all file fields (and in this case, I will explain later in the documentation how to change the default format) or do I add the tabular format?

tabular only is totally ok I think. The questions was more how does your tool decide between tabular and csv? Can it support both at the same time?

I based myself on my R shiny app: you have to choose the column separator of the file. But if the csv (or txt) are a problem, I remove it.

bgruening · 2020-09-29T20:04:25Z

tools/rgcca/rgcca.xml

+            #set $data_paths += ',' + $s.block_n.file_name
+            #set $data_names += ',' + $s.block_n.name
+        #end for
+        Rscript $__tool_directory__/launcher.R


every path should be single quoted.

bgruening · 2020-09-29T20:05:43Z

tools/rgcca/rgcca.xml

+        Rscript $__tool_directory__/launcher.R
+            --datasets '${$data_paths}'
+            --names '${$data_names}'
+            --directory $__tool_directory__


what does this directory do? Please note that this directory is read only.

remnants of codes from an old version without conda. removed.

bgruening · 2020-09-29T20:05:54Z

tools/rgcca/rgcca.xml

+            #set $data_names += ',' + $s.block_n.name
+        #end for
+        Rscript $__tool_directory__/launcher.R
+            --datasets '${$data_paths}'


this looks like a $ too many

ecamenen · 2020-10-01T16:24:27Z

@bgruening if I have understood everything correctly, all the modifications requested by your review have been made (remaining: GTN; optional: tabular format).

ecamenen · 2020-10-14T12:17:52Z

hi @bgruening. i've made the last changes ;)

bgruening

Can you remove the files under static/images/ please? They do not seems to be used.

Sorry, I have very limited time lately. @bernt-matthias could you help here please?

bgruening · 2020-11-16T13:45:28Z

tools/rgcca/rgcca.xml

+            #set $data_paths += ',' + $s.block_n.file_name
+            #set $data_names += ',' + $s.block_n.name
+        #end for
+        Rscript $__tool_directory__/launcher.R


bgruening · 2020-11-16T13:50:28Z

tools/rgcca/rgcca.xml

+    <inputs>
+        <param name="block_1" type="data" format="tsv" label="Load a block"
+            help="TSV file containing a matrix with: (i) quantitative values only (decimal should be separated by '.'), (ii) the samples in lines (should be labelled in the 1rst column) and (iii) variables in columns (should have a header)."/>
+        <repeat name="dataset" title="New block"


can you simply use a <param name="block_n" type="data" format="tsv" multiple="true" optional="true" label="Dataset"/> and remove the repeat?

Or even better, merge block_1 and block_n and make it <param name="block_n" type="data" format="tsv" multiple="true" optional="false" label="Dataset"/>

@bernt-matthias Sorry for the delay, in the meantime my contract was finished and I don't had much time for new features.
I don't know about multiple data, I think it's less intuitive for biologist users in my lab but if you want I can code that. [For example, I don't know how to select only certain data (Cf. image) and suppress the ones we don't want to. So the biologist don't either.]
As a result, I can't get my tool to work with multiple data anymore with this notation. I looked at the help sections I could find and the example (https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/multi_data_param.xml) but I m more a R programmer. Can you help me with that? I pushed a broken version for the example. The test on a single file works, but not multiple files.

@bernt-matthias Sorry for the delay, in the meantime my contract was finished and I don't had much time for new features.

No worries.

I don't know about multiple data, I think it's less intuitive for biologist users in my lab but if you want I can code that. [For example, I don't know how to select only certain data (Cf. image) and suppress the ones we don't want to. So the biologist don't either.]

You can add more data sets by holding the Ctrl key while clicking data sets in the selcetor. You can select complete sets of files by holding Shift and click.

As a result, I can't get my tool to work with multiple data anymore with this notation. I looked at the help sections I could find and the example (https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/multi_data_param.xml) but I m more a R programmer. Can you help me with that? I pushed a broken version for the example. The test on a single file works, but not multiple files.

For the test you can simple use a comma separated list of file named in the parameter.

#set data_paths = ",".join([_.file_name for _ in $blocks]) #set data_names = ",".join([_.name for _ in $blocks])

might be a short cut to the for loop. But not sure if it works.

bernt-matthias · 2020-11-16T13:58:09Z

tools/rgcca/macro.xml

+
+    <token name="@TOOL_VERSION@">3.0.0</token>
+
+    <token name="@BLOCK_RULES@">0 corresponds to the superblock (or the last block loaded), 1 to the first block, 2 to the second one, etc. This number should not be greater than the number of blocks selected.</token>


Maybe add the default, as in COMP_RULES

bernt-matthias · 2020-11-16T14:05:11Z

tools/rgcca/rgcca.xml

+    <inputs>
+        <param name="block_1" type="data" format="tsv" label="Load a block"
+            help="TSV file containing a matrix with: (i) quantitative values only (decimal should be separated by '.'), (ii) the samples in lines (should be labelled in the 1rst column) and (iii) variables in columns (should have a header)."/>
+        <repeat name="dataset" title="New block"


Or even better, merge block_1 and block_n and make it <param name="block_n" type="data" format="tsv" multiple="true" optional="false" label="Dataset"/>

bernt-matthias · 2020-11-16T14:09:07Z

tools/rgcca/rgcca.xml

+    </inputs>
+
+    <outputs>
+        <data name="individual_plot" label="individuals.pdf" format="pdf"/>


please add labels to the outputs that distinguishes them eg ${tool.name} on ${on_string}: individual plot

are there some optional outputs? then you may add a select to the inputs and make the outputs conditional on the users choice.

BTW, do you know how to fix this ('data_2 and data_1')?

Looks good to me. What exactly would you like to fix?

Maybe there is a way to display the real name of the data files (e.g. agriculture.txt and industry.txt)? data2 and data1 doesn't look great. Otherwise, I'll take on ${on_string} off.

I would strongly vote for leaving it as is, because that's Galaxy's default naming scheme.

bernt-matthias · 2020-11-16T14:10:26Z

tools/rgcca/rgcca.xml

+            </conditional>
+
+            <conditional name="tau">
+                <param name="bool" type="boolean" truevalue="optimal" falsevalue="false" checked="false" label="Use an optimal tau"


By the IUC standards this should be a select.

bernt-matthias · 2020-11-16T14:15:41Z

tools/rgcca/rgcca.xml

+        <requirement type="package" version="@TOOL_VERSION@">rgccacmd</requirement>
+    </requirements>
+
+    <stdio>


You may just add detect_errors="exit_code" to the command tag instead

bernt-matthias · 2020-11-16T14:16:20Z

tools/rgcca/macro.xml

+    <token name="@COMP_RULES@">This number should not be greater than the selected number of component (2, by default).</token>
+
+    <xml name="output_tests" token_path="" token_compx="1" token_compy="2">
+        <param name="block_1" value="agriculture.tsv" />


please add ftype

bernt-matthias · 2020-11-16T14:16:37Z

tools/rgcca/macro.xml

+
+    <xml name="output_tests_3blocks">
+        <repeat name="dataset">
+            <param name="block_n" value="industry.tsv"/>


bernt-matthias · 2020-12-06T19:47:02Z

tools/rgcca/launcher.R

@@ -0,0 +1,528 @@
+# Author: Etienne CAMENEN


It appears to me that you copied this file from https://github.com/rgcca-factory/RGCCA/blob/master/inst/launcher.R which is for version 3.0? Do you know for sure if it works also for 2.1.2?

Would it be an option if I try to update the conda package to 3.0 and include the launcher directly there. Then we do not need to have it here.

The galaxy will not work with the latest version of CRAN which is 2.1.2. I worked with statisticians on the github, we had to make concessions and modifications and it's not always very clean :P We can talk about a pre-version tagged "3.0.0" that I packaged in its current conda (and finally, we decided to name it rather 2.9.0). But it's not yet available for the CRAN (WIP).

Not knowing in advance how Galaxy worked with Conda and learning little by little, I actually had to make a "patch" by copying and modifying a recent version of my command line interface (launcher.R) for this package (bioconda/bioconda-recipes#22918 (comment)).

You can indeed take the previous version of my conda to update the launcher with what your lintr asked for and change the path so that I can call it from the conda (I'm pretty new in conda). I'm used to put R-Shiny files and external scripts (as launcher.R) in an "inst" folder, but from memory I hadn't managed to access the CONDA_PREFIX variable via Galaxy. Hence my "copy and paste" patch.

tools/rgcca/rgcca.xml

Co-authored-by: M Bernt <m.bernt@ufz.de>

ecamenen · 2020-12-07T17:53:28Z

        #set data_paths = ",".join([str(_.file_name) for _ in $blocks])
        #set data_names = ",".join([str(_.name) for _ in $blocks])
Note that since data set and file names may contain commas this may fail. We could search and replace , by _

I didn't understand too much. And it will make me change internal functions, tests and documentation on several deliverables derived from this RGCCA package (docker, shiny, command line). We have a lot of files with "_" in our lab. Under R, the file existence checks have been performed with the file.exists() function (https://github.com/rgcca-factory/RGCCA/blob/shiny/R/check_file.R).

bernt-matthias · 2020-12-07T18:37:20Z

        #set data_paths = ",".join([str(_.file_name) for _ in $blocks])
        #set data_names = ",".join([str(_.name) for _ in $blocks])
Note that since data set and file names may contain commas this may fail. We could search and replace , by _
I didn't understand too much. And it will make me change internal functions, tests and documentation on several deliverables derived from this RGCCA package (docker, shiny, command line). We have a lot of files with "_" in our lab. Under R, the file existence checks have been performed with the file.exists() function (https://github.com/rgcca-factory/RGCCA/blob/shiny/R/check_file.R).

Actually only the data_names are a problem. Assume that you have a data set with a , in the name. Then the splitting will be wrong. But we can fix this on the Galaxy side. I will suggest a fix once the tests are passing.

bernt-matthias · 2020-12-07T18:38:54Z

Cool tests are passing :). Can the messages:

Warning message:
In if (class(y) %in% c("matrix", "data.frame")) { :
  the condition has length > 1 and only the first element will be used

be ignored?

tools/rgcca/rgcca.xml

bernt-matthias · 2020-12-07T18:50:58Z

tools/rgcca/rgcca.xml

+    </inputs>
+
+    <outputs>
+        <data name="individual_plot" label="${tool.name} on ${on_string}: individuals.pdf" format="pdf"/>


This is how you use the output selector in the outputs:

Suggested change

<data name="individual_plot" label="${tool.name} on ${on_string}: individuals.pdf" format="pdf"/>

<data name="individual_plot" label="${tool.name} on ${on_string}: individuals.pdf" format="pdf">

<filter>"individuals" in output_selector</filter>

</data>

bernt-matthias · 2020-12-07T18:53:36Z

tools/rgcca/rgcca.xml

+
+        <test expect_num_outputs="8" expect_exit_code="0">
+            <expand macro="output_tests" path="1block"/>
+            <param name="blocks" value="agriculture.tsv" ftype = "tsv"/>


Also in the tests you can then select the desired outputs:

Suggested change

<param name="blocks" value="agriculture.tsv" ftype = "tsv"/>

<param name="blocks" value="agriculture.tsv" ftype = "tsv"/>

<param name="output_selector" value="individual,corcircle,top_variables">

and you would need to change expect_num_outputs="3"

ecamenen · 2020-12-08T14:25:47Z

        #set data_paths = ",".join([str(_.file_name) for _ in $blocks])
        #set data_names = ",".join([str(_.name) for _ in $blocks])
Note that since data set and file names may contain commas this may fail. We could search and replace , by _
I didn't understand too much. And it will make me change internal functions, tests and documentation on several deliverables derived from this RGCCA package (docker, shiny, command line). We have a lot of files with "_" in our lab. Under R, the file existence checks have been performed with the file.exists() function (https://github.com/rgcca-factory/RGCCA/blob/shiny/R/check_file.R).
Actually only the data_names are a problem. Assume that you have a data set with a , in the name. Then the splitting will be wrong. But we can fix this on the Galaxy side. I will suggest a fix once the tests are passing.

Ah ok, I didn't understand! It's managed internally in an R function, but if it's done in front-end, that's great.

ecamenen · 2020-12-08T14:26:45Z

Cool tests are passing :). Can the messages:

Warning message:
In if (class(y) %in% c("matrix", "data.frame")) { :
  the condition has length > 1 and only the first element will be used

be ignored?

If I remember where it is, it's an internal function that has been fixed since then. If a new conda is made, I suppose we can start on the version under development (the "shiny" branch, let's say) which is the latest one in progress and which should not know any major modifications.
Note for me : check_size_blocks.R l5:
if (any(class(y) %in% c("matrix", "data.frame"))) {

Co-authored-by: M Bernt <m.bernt@ufz.de>

ecamenen · 2020-12-08T16:13:51Z

I have not closed comments, but normally all suggestions made more than two days ago have been made.

bernt-matthias

@bgruening ready for merge?

ecamenen · 2021-02-16T09:36:43Z

Hello @bernt-matthias, we would like to present our galaxy tool in congress. I thought that for future updates the tool would appear in the toolbar here https://usegalaxy.eu/ , but it doesn't seem to be the case. Can you tell me how to access it from "your official server" ?

bernt-matthias · 2021-02-16T09:39:26Z

Hi @ecamenen you need to add it to this list: https://github.com/usegalaxy-eu/usegalaxy-eu-tools/blob/master/tools_iuc.yaml by opening a PR over there.

ecamenen added 3 commits September 2, 2020 17:03

added rgcca tool

f765321

formatting launcher.R, removed comments, superfluous whitespace, glob…

21bc184

…al visibility

text parameters 'single quoted', --connection instead of -c

f480f05

bernt-matthias added the new software label Sep 17, 2020

ecamenen added 2 commits September 29, 2020 16:43

wrong spaces for lintr, global variables

60c7e9f

removed useless ' in xlm command, has_text sucess

78b698a

bgruening reviewed Sep 29, 2020

View reviewed changes

ecamenen added 5 commits September 30, 2020 12:13

variables small letters, tsv connection, single quoted paths

096c7e6

TSV default for all inputs, contact modified

b713359

added tsv tests

1233ff9

removed directory option

85e8744

removed useless boolean

2c966a9

ecamenen added 3 commits October 2, 2020 14:39

removed parsing section

34d2809

updated tutorial

84b0242

removed tutorial, added tutorial link

d4726ae

bgruening reviewed Nov 16, 2020

View reviewed changes

bernt-matthias reviewed Nov 16, 2020

View reviewed changes

ecamenen added 3 commits November 18, 2020 18:46

added default block in label, added name for pdf labels, removed static

ec738b6

modified tau as select option

27696f0

multiple data inputs

42e7aad

bernt-matthias reviewed Dec 6, 2020

View reviewed changes

tools/rgcca/rgcca.xml Outdated Show resolved Hide resolved

bernt-matthias reviewed Dec 6, 2020

View reviewed changes

tools/rgcca/rgcca.xml Outdated Show resolved Hide resolved

ecamenen and others added 3 commits December 7, 2020 17:09

Update tools/rgcca/rgcca.xml

fd84be0

Co-authored-by: M Bernt <m.bernt@ufz.de>

Update tools/rgcca/rgcca.xml

ea46a48

Co-authored-by: M Bernt <m.bernt@ufz.de>

updated data parsing command line, debugged tests

9b77971

added multiple input formats, parsing section

cb4484b

bernt-matthias reviewed Dec 7, 2020

View reviewed changes

tools/rgcca/rgcca.xml Outdated Show resolved Hide resolved

bernt-matthias reviewed Dec 7, 2020

View reviewed changes

ecamenen and others added 3 commits December 8, 2020 15:28

data_name spliting fix tools/rgcca/rgcca.xml

41b70e8

Co-authored-by: M Bernt <m.bernt@ufz.de>

added a filter for outputs

4f49310

Co-authored-by: M Bernt <m.bernt@ufz.de>

filtered outputs + tests

06482d4

bernt-matthias approved these changes Dec 9, 2020

View reviewed changes

bgruening approved these changes Jan 12, 2021

View reviewed changes

bgruening merged commit 00f9e92 into galaxyproject:master Jan 12, 2021

ecamenen mentioned this pull request Mar 9, 2021

Add rgcca in tools_iuc.yml usegalaxy-eu/usegalaxy-eu-tools#372

Merged


		<token name="@TOOL_VERSION@">3.0.0</token>

		<token name="@BLOCK_RULES@">0 corresponds to the superblock (or the last block loaded), 1 to the first block, 2 to the second one, etc. This number should not be greater than the number of blocks selected.</token>

-        <data name="individual_plot" label="${tool.name} on ${on_string}: individuals.pdf" format="pdf"/>
+        <data name="individual_plot" label="${tool.name} on ${on_string}: individuals.pdf" format="pdf">
+            <filter>"individuals" in output_selector</filter>
+        </data>

	<param name="blocks" value="agriculture.tsv" ftype = "tsv"/>
	<param name="blocks" value="agriculture.tsv" ftype = "tsv"/>
	<param name="output_selector" value="individual,corcircle,top_variables">

Add rgcca #3179

Add rgcca #3179

Conversation

ecamenen commented Sep 2, 2020 • edited Loading

bernt-matthias commented Sep 17, 2020

ecamenen commented Sep 29, 2020

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecamenen Oct 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecamenen commented Oct 1, 2020

ecamenen commented Oct 14, 2020

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bernt-matthias Dec 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecamenen Dec 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecamenen commented Dec 7, 2020

bernt-matthias commented Dec 7, 2020

bernt-matthias commented Dec 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecamenen commented Dec 8, 2020

ecamenen commented Dec 8, 2020 • edited Loading

ecamenen commented Dec 8, 2020

bernt-matthias left a comment

Choose a reason for hiding this comment

ecamenen commented Feb 16, 2021

bernt-matthias commented Feb 16, 2021

ecamenen commented Sep 2, 2020 •

edited

Loading

ecamenen Oct 1, 2020 •

edited

Loading

bernt-matthias Dec 4, 2020 •

edited

Loading

ecamenen Dec 8, 2020 •

edited

Loading

ecamenen commented Dec 8, 2020 •

edited

Loading