Tool form section tag

[guerler]

This adds a section grouping tag to tool form parameters as discussed in https://trello.com/c/KxlQK0FB/587-ui-tools-provide-methods-for-visually-grouping-parameters.

The format is: section name="NAME" title="TITLE" expanded="True/False"

[dannon]

This is an automated message. Thanks for your contribution, a Trello card to track this issue has been created. Apply this patch for testing.

[jmchilton]

Opened PR to your branch implementing tool testing functionality.

[guerler]

Awesome. Thx.

[jmchilton]

A couple things - this looks like this will end the old tool form? I am nervous about that because we haven't implemented an upload widget for the new tool form - right?

Also have you checked out templates/webapps/tool_shed/repository/tool_form.mako - you may want to issue a warning or something there that the tool uses components not displayed. Ultimately it would be awesome to use the new tool form there as well - how hard to you think it would be to render the form without runtime history data and a Galaxy API?

[guerler]

Thx for the comments. I think that the upload tool parameter should be deprecated since it is not a clean solution. Instead users should upload files through the regular UI. Regarding the tool shed form mako, the best solution is probably to extend the current mako and then later this year consider to rewrite the tool shed UI without makos at all.

[martenson]

We can also remove the form-preview functionality from the TS and just redirect (or show a modal) to a specialized usegalaxy.org/preview?tool_id=xxxxxxxx service.

[martenson]

👍

[dannon]

@guerler -- This disables the old tool form completely, removing the 'toolform_upgrade' conditional from tool_form.mako. Is that intentional, or is it left over from your testing?

[dannon]

(the line in question, per my comment below -- intentional?)

[guerler]

Yes, this disables the old tool form for good. Currently we are building both tool forms in parallel. Since sections are not supported in the old tool form, we can either integrate them or disable the old tool form for good which would allow us to clean up the code and remove the associated makos all together. However, as John pointed out the latter has the consequence that the upload and ftp tool parameters will be deprecated.

[dannon]

Ahh, got it -- I hadn't put it together that John was talking about this when he said 'end the old tool form', somehow. Super short deprecation warning, we probably need to handle this and not just purge it all quite yet?

[dannon]

Discussed with @guerler offline a bit, but this can't go as-is since it removes capabilities that still need to be reproduced in the new tool form, and it disables the old one. Path forward is going to be to implement the missing features in the new tool form so there's no functionality loss and everything is fully backwards compatible, and then (and only then) disable the old tool form and remove it from the codebase.

[jmchilton]

So this PR made me nervous - but this statement "no functionality loss and everything is fully backwards compatible" also makes me nervous in the other direction. Is composite uploads what need to be preserved or is having a upload widget that can be embedded a tool?

[dannon]

I think what we were leaning towards was an upload widget. @guerler -- want to comment w/ plans?

[jmchilton]

I really have very mixed feelings - but if the intention is to continue to support composite uploads - we should restore a link to the old upload form on main - because currently there is no link and we are telling people these datatypes are no longer supported https://biostar.usegalaxy.org/p/11371/. No reason to lose the goodwill now and then have to support these indefinitely anyway :).

[bgruening]

@jmchilton this biostar link was hopefully only referring to PLED files and the tools developed at this time. We need to upload composite datatypes also in GalaxyP - wiff - scream, run, hide!

[jmchilton]

Believe me @bgruening I want a usable wiff datatype in Galaxy - I am just not sure two or three datatypes warrant another 1 or 2 months of @guerler's time. They may be - I really have no clue though.

[jxtx]

I think composite datatypes are important.

[jmchilton]

Also worth reading... https://www.mail-archive.com/galaxy-dev@lists.galaxyproject.org/msg00024.html

[bgruening]

@jmchilton this I can not judge. Just wanted to point out that there are some rare cases.

[jmchilton]

@jxtx Did you intend to say "uploading of composite datatypes are important" - that is the stakes of this - not removing them completely.

[jxtx]

I think it is probably worth spending the effort to make them uploadable now.

[jmchilton]

@jxtx Awesome - if you think it is time well spent - my conscience is clear.

[guerler]

So it seems that we have two issues here: 1. the composite uploads and 2. the upload/ftp tool parameter. For the composite uploads I wonder if we can provide a tool which combines files uploaded through the regular upload modal into composite datatypes? Regarding the upload/ftp tool parameter (which is only used in one case within the toolshed) I would look out for a workaround i.e. by simply replacing them with regular dataset selectors and asking the user to use the regular upload modal maybe?

[guerler]

Regarding 2.: Since no one in the toolshed (one outcommented attempt excluded) is using neither the file nor the ftpfile tool parameters, we actually might want to deprecate them. The alternative would be to fix and rewrite them such that they properly work incl. in the workflow editor. Instead we could ask the community if anyone is using these parameter types outside the tool shed. Depending on the number of responses, users should be able to simply adjust their tools by using regular dataset selectors.

[guerler]

Since this seems to be a separate discussion. I have re-activated the tool form fall back option for this PR.

[jmchilton]

Awesome - thanks @guerler.

[natefoo]

@guerler - For the composite uploads I wonder if we can provide a tool which combines files uploaded through the regular upload modal into composite datatypes

This seems like a pretty reasonable way to do it that'll fall in line with the way collections are built.

[blankenberg]

Composite datasets and collections are different in so many ways, allowing upload of the individual parts and then combining them is probably not feasible, since it is quite possible for the individual parts to not be uploadable e.g., random generic "binary content not allowed", whereas if uploaded as the composite datatype the individual files together are valid for the datatype.

Also, the case of uploading a composite datatype piecemeal will waste disk space/usage, and part of the reason that we have composite datasets in many cases is that the individual items "are not useful by themselves".

Also, any tool to combine them will need to use the same/similar underlying logic as the current uploading of these -- that is allowing setting metadata, modifying the names of files, etc.

[blankenberg]

In reference to no longer supporting certain input types, the ToolShed is not the end-all-be all of tool availability. Many tools written by individuals may not ever make it to the ToolShed, but that doesn't mean they should stop working, or that we should force them to change their tool to use different parameter types.

We have also stated in the past that we would maintain backwards compatibility, specifically with respect to tool definitions.

[jmchilton]

Point well articulated on composite datatypes - I was trying to explain that last week out-of-band but you did a much better job.

As for preserving type="file" indefinitely with the exact same API - that is worrying to me. I know from experience that modifying upload.xml to do non-trivial things requires modifications to upload.py. Do you also consider everything in upload.py part of Galaxy's stable API? Is any modified upload.py suppose to work forever against any future version of Galaxy? Here are the Galaxy imports it makes -

from galaxy import eggs
# need to import model before sniff to resolve a circular import dependency
import galaxy.model
from galaxy.datatypes.checkers import *
from galaxy.datatypes import sniff
from galaxy.datatypes.binary import *
from galaxy.datatypes.images import Pdf
from galaxy.datatypes.registry import Registry
from galaxy import util
from galaxy.datatypes.util.image_util import *
from galaxy.util.json import *

From the outside, my impression was that type="file" was never a serious part of the tool API - it was bolted on to allow a very specific kind of upload process to work as tools. If we want to get serious about allowing customization to the upload process that is great - lets do that within the new framework - not try to preserve the old API that we don't even know is being used anywhere.

[natefoo]

From the outside, my impression was that type="file" was never a serious part of the tool API - it was bolted on to allow a very specific kind of upload process to work as tools.

This is right.

[blankenberg]

Unfortunately, they are listed on https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax

type="color"

(New as of 15.03 release.) ...

type="genomebuild"

...

type="hidden"

...

type="hidden_data"

...

type="baseurl"

...

type="file"

...

type="ftpfile"

...

type="library_data"

... 

[jmchilton]

I think the entirety of its documentation being literally ... is a very good indication that it wasn't a serious part of the API.

[blankenberg]

This documentation is poor and we've gotten quite better at documenting things, but I'd be remiss to not point out that some documentation does exist.

That being said, I don't actually like these parameters to be used by 'non-built in' tools (e.g. upload). Tools should be working on Datasets and not uploaded files.

[natefoo]

I suspect they were only placed on that page because someone pulled every possible value out with the intent that they should be addressed somehow at some point. However, I don't believe we ever intended for any of those file params to be used directly in a tool.

[guerler]

This is a pretty good discussion. Thx for all the input.