Rename shed-install to shed-tools install and add --latest and --revisions flag. Add shed-tools update subcommand. #64

rhpvorderman · 2017-11-24T13:12:00Z

Solves #63

Shed-install was diversified into an install and update command. The syntax now is changed.:

To install: shed-install [flags] -> shed-tools install [flags]
To update: shed-tools update

This uses built in functionality:

~~shed-install~~ shed-tools install already defaulted to the latest installable revision when no revision is given for a tool. The --latest flag forces this behaviour on all tools in the tool list. The latest revision will be installed regardless of the revision in the tool list.
~~shed-install --update~~ shed-tools update now uses get_tool_list_from_galaxy to get a tool list from the existing galaxy as a dictionary. Then it does the same as shed-tools install --latest.

To test all three possible methods of tool specifying for ~~shed-install~~ shed-tools install, --revisions was added to allow revisions installation from the command line.

Some other changes had to be made to enable this.

refactor ~~shed-install~~ shed-tools variables to be more readable
move some common methods out of ~~shed-install~~ shed-tools to reduce the file length. This includes the logging
write extra tests
get_tool_list now does not write a file on instantiation anymore, but instead creates a class that can write a file or just simply give the dictionary.

Other nice changes that resulted from the refactoring:

run-data-managers and ~~shed-install~~ shed-tools now use the same logging method. Logging can now also be extended to other tools.
the get_galaxy_connection function was updated so it can optionally also use a file (i.e. a tool_list) to determine connection parameters.
get_tool_list now uses bioblend api instead of custom connection.
A check_url method checks the galaxy_url now for get_tool_list and run-data-managers as well broadening the scope of #49

EDIT: 2017-11-27, add changes to reflect further changes.
EDIT: 2017-12-27, clarify that --latest works on the whole tool list.

The text was updated successfully, but these errors were encountered:

…og issue" This reverts commit ab6723f.

rhpvorderman · 2017-11-24T13:43:07Z

@jmchilton It would be nice if it could be included in the following ephemeris release. #61 . After someone reviews it of course.

mvdbeek

This looks really good!

mvdbeek · 2017-11-24T13:59:54Z

ephemeris/shed_install.py

-    itl = tsc.get_repositories()
-    for it in itl:
-        if it['status'] == 'Installed':
+    installed_tool_list = tool_shed_client.get_repositories()


Now that you're at it, can we change this to installed_repository_list, to avoid any confusion, and to leave some wigglespace should we ever implement a more tool-centric installation approach?

In fact I think you can s/tool/repository/g and s/tools/repositories/g in this file, this should be much clearer.

rhpvorderman · 2017-11-24T15:34:20Z

Done with all the refactoring.

jmchilton · 2017-11-24T16:35:08Z

Oh wow - this is fantastic!

bgruening

Impressive work @rhpvorderman! I have included a few comments inline.

bgruening · 2017-11-24T22:02:45Z

ephemeris/shed_install.py

                tool panel that were installed on the target Galaxy instance
                from the Tool Shed;
-                - `tool_panel_custom_tools` with a list of tools available in
+                - `tool_panel_custom_tools` with a list of repositories available in


repositories should here be tool, right?

Yes. I did some find -> replace (in selection) stuff. It worked better than just doing it for the entire file. But I still missed some things. Thanks for pointing them out.

bgruening · 2017-11-24T22:02:45Z

ephemeris/shed_install.py

    """
    if not omit:
        omit = []
-    tp_tools = []  # Tools available in the tool panel and installe via a TS
+    tool_panel_repos = []  # The repositories from tools available in the tool panel and installable via a TS


s/from/of/

maybe the 'the Tool Shed repositories of tools available ...'

bgruening · 2017-11-24T22:02:45Z

ephemeris/shed_install.py

+                        default=None,
+                        dest="revisions",
+                        help="The revisions of the tool repository that will be installed. "
+                             "Revisions must be specified in on the command line in YAML format:"


s/in//

But this is rather odd for the UX. Can we use nargs="" here instead?

bgruening · 2017-11-24T22:02:45Z

ephemeris/shed_install.py

+    parser.add_argument("--update",
+                        action="store_true",
+                        dest="update_tools",
+                        help="This updates all tools in the Galaxy to the latest revision. "


I'm wondering if we should make this a positional command.

Like shed_tools update and shed_tools install ...

bgruening · 2017-11-24T22:02:45Z

tests/test.sh

+
+function start_new_container {
+    echo "Start new container"
+    docker rm -f $CID


Interesting will this automatically stop the container? Or should we stop the container before?

rm -f automatically stops the container. That is what the -f flag is for.

bgruening · 2017-11-24T22:02:45Z

tests/test.sh

+echo "Check tool installation with yaml on the commandline"
+# CD Hit was chosen since it is old and seems to be unmaintained. Last update was 2015.
+# Anyone know a smaller tool that could fit its place?
+OLD_TOOL="{'owner':'jjohnson','name':'cdhit','revisions':['34a799d173f7'],'tool_panel_section_label':'CD_HIT'}"


We should probably create a dummy tool in the TS. As soon as this container gets updated we will have failing tests that are hard to understand.

I agree. Better even to use the test toolshed for this.

No, this is fine, the test toolshed is not a good idea. We do similar stuff in galaxy's tests, and even if the repository get's updated this will continue to work.

actually we can just make sure that there is a new revision, we don't care which revision that is.

@mvdbeek correct me, but if we test the latest repository as we do a few lines below, this test will fail as soon as the repo is updated.

right, but we can test that there is an additional revision in the tool list

right, but we can test that there is an additional revision in the tool list

Additional revisions are only installed if there are multiple versions. If the revision is of the same version, then it will override the previous revision and not show up with get-tool-list.

CD-HIT was chosen because it was last updated more than 2 years ago. The last update before that was 5 years ago. It seems to be pretty stable. If it changes, it is not too hard to figure out why the tests fail.

If we really want to do this right, we use a docker image to start up a custom toolshed, upload a custom tool to it, and then test it. But I am not sure this is worth the effort. Personally I think the test as it is now is sufficient.

This avoids iterating over the same listy twice and deleting items from the list. Also should be more robust to empty sections.

mvdbeek · 2017-11-26T17:47:51Z

@rhpvorderman I added some minor tweaks here: rhpvorderman#2

This works really well for me!

Minor tweaks to robustness of shed-install --update

rhpvorderman · 2017-11-27T08:03:37Z

Will now work on making update a subcommand.

rhpvorderman · 2017-11-27T11:58:44Z

Done with everything except the dummy tool. I do not have time to work on this though. (I only have uploaded one tool in the toolshed and that was some time ago, so this will take quite some time for me to get right again) Can somebody else pick this up? Or we could file an issue and leave this for a later date.

I have altered the message at the top to reflect the further changes.

bgruening · 2017-12-22T12:42:23Z

This is a great Christmas present for the Galaxy community. Thanks a bunch @rhpvorderman!

xref: #66

martenson · 2017-12-22T21:00:22Z

Simply great, awesome contribution @rhpvorderman, thank you!

rhpvorderman added 25 commits Nov 23, 2017

refactor some variables for readabality

e164a59

move logging methods to other file

7d9a269

move common methods out of shed_install

189f351

clearer names refactoring

93a0ddb

pep8 issues

e82bb92

add --latest flag

999d52e

move logging back into shed-install because of weird double log issue

ab6723f

Revert "move logging back into shed-install because of weird double l…

b55e4bd

…og issue" This reverts commit ab6723f.

fix latest add test

a6cf769

added revisions option

78bd402

Tests added for shed-install

239c78a

fixed some errors

154ae16

added update flag

2a726b1

added test

a93e772

shed-install also uses main()

e573868

added method to get tool_panel

06c3ed8

tinny bug

1f5c3f7

rewrite get_tool_list to be used by shed install and in other methods

851f772

rewrite get tool list to use bioblend api

e406033

logging methods now common

094fe2c

some bugs fixed

06b83cb

flake issues

1aad702

proper galaxy connection integration

b4e373c

typo

16495b9

fixed double log issue

caf28c4

mvdbeek reviewed Nov 24, 2017

View changes

rhpvorderman added 2 commits Nov 24, 2017

refactor tools into repos where applicable

0e83ce3

flake8 issues

ca4a06f

rhpvorderman added 2 commits Nov 24, 2017

solve double logging issue

75e3d15

remove global

808ea70

rhpvorderman force-pushed the GALAXY-93 branch 2 times, most recently from caec176 to 808ea70 Nov 24, 2017

rhpvorderman closed this Nov 24, 2017

rhpvorderman reopened this Nov 24, 2017

bgruening reviewed Nov 24, 2017

View changes

mvdbeek added 2 commits Nov 25, 2017

Break out of while loop when finished is True

3610e1c

Simplify merging of repositories

52b0b7e

This avoids iterating over the same listy twice and deleting items from the list. Also should be more robust to empty sections.

rhpvorderman added 2 commits Nov 27, 2017

use nargs for revisions some textual updates

87861e6

Merge pull request #2 from mvdbeek/GALAXY-93

61d04e1

Minor tweaks to robustness of shed-install --update

rhpvorderman added 5 commits Nov 27, 2017

shed-install renamed to shed-tools

7e7937c

seperate out shed-tools install and shed-tools update in the parser

ff9ffa7

update documentation to reflect changes

da40dcf

Documentation updated

728ce2f

remove docker restarting code

20bef78

rhpvorderman mentioned this pull request Nov 27, 2017

0.7.1 Release #61

Closed

rhpvorderman changed the title ~~Add --latest flag , --update flag and --revisions flag to shed-install.~~ Rename shed-install to shed-tools install and add --latest and --revisions flag. Add shed-tools update subcommand. Nov 27, 2017

bgruening mentioned this pull request Dec 22, 2017

create a frozen TTS repo for testing updates and such #66

Open

bgruening merged commit cb1d49c into galaxyproject:master Dec 22, 2017
1 check passed

Loading

rhpvorderman deleted the GALAXY-93 branch Dec 28, 2017

galaxyproject / ephemeris Public

Rename shed-install to shed-tools install and add --latest and --revisions flag. Add shed-tools update subcommand. #64

Rename shed-install to shed-tools install and add --latest and --revisions flag. Add shed-tools update subcommand. #64

rhpvorderman commented Nov 24, 2017 •

edited

Loading

rhpvorderman commented Nov 24, 2017

mvdbeek left a comment

mvdbeek Nov 24, 2017

rhpvorderman Nov 24, 2017

rhpvorderman commented Nov 24, 2017

jmchilton commented Nov 24, 2017

bgruening left a comment

bgruening Nov 24, 2017

rhpvorderman Nov 27, 2017

bgruening Nov 24, 2017

bgruening Nov 24, 2017

bgruening Nov 24, 2017

bgruening Nov 24, 2017

bgruening Nov 24, 2017

rhpvorderman Nov 27, 2017

bgruening Nov 24, 2017

rhpvorderman Nov 27, 2017

mvdbeek Nov 27, 2017 •

edited

Loading

mvdbeek Nov 27, 2017

bgruening Nov 27, 2017

mvdbeek Nov 27, 2017

rhpvorderman Nov 28, 2017

mvdbeek commented Nov 26, 2017

rhpvorderman commented Nov 27, 2017

rhpvorderman commented Nov 27, 2017

bgruening commented Dec 22, 2017

martenson commented Dec 22, 2017

galaxyproject / ephemeris Public

Rename shed-install to shed-tools install and add --latest and --revisions flag. Add shed-tools update subcommand. #64

Rename shed-install to shed-tools install and add --latest and --revisions flag. Add shed-tools update subcommand. #64

Conversation

rhpvorderman commented Nov 24, 2017 • edited Loading

rhpvorderman commented Nov 24, 2017

mvdbeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhpvorderman commented Nov 24, 2017

jmchilton commented Nov 24, 2017

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek Nov 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek commented Nov 26, 2017

rhpvorderman commented Nov 27, 2017

rhpvorderman commented Nov 27, 2017

bgruening commented Dec 22, 2017

martenson commented Dec 22, 2017

rhpvorderman commented Nov 24, 2017 •

edited

Loading

mvdbeek Nov 27, 2017 •

edited

Loading