Done: Refactor shedtools #104

rhpvorderman · 2018-07-18T07:30:15Z

~~After two attempts refactoring the existing file, I am going to refactor shed tools from scratch. I will use the issue #91 as a guideline.~~

~~Please do not add any functionality to shed tools before the refactoring complete. I will be pushing regularly so you can check on my progress. I will also be available on gitter.~~

Done.
The following stuff has changed:

shed_tools arguments and methods now in a separate file (for readability reasons)
InstallToolManager rewritten to allow for a much cleaner interface. No storing of variables in self that are used by only one method. InstallToolManager can now be instantiated with only a GalaxyInstance. Methods also have a sane number of arguments. So, when people where referencing shed_tools.py in other scripts, these scripts are now broken. On the upside, referencing shed_tools in other scripts is now much easier
Global log was removed
Complexity dropped by 3 units. (Whatever that is supposed to mean)
Useless stuff was chucked out. Some defaults were passed through functions 5 times.
get-tool-list can now get all the revisions of each tool, not just the revisions that are in the tool panel. Admin rights are needed for this. Shed-tools uses this functionality to get the installed tools.
Whether tools are already installed is now determined by getting a list of the installed tools and checking that. The previous method did this via checking every tool with an API call to galaxy. This was of course slow. Now it should be faster (even though the checking method is gruesomely inefficient, but it should be faster than doing ssl handshakes all the time).
shed-tools update now also accepts tool lists. It will only update tools that are already installed on the galaxy and issue a warning for other tools.
The --skip_install_tool_dependencies flag is no longer functional as this is the default. Same goes for the --install_resolver_dependencies flag. These flags are still around for backwards compatibility, but do not do anything. --install_tool_dependencies, --skip_install_resolver_dependencies and --skip_install_repository_dependencies where added.
Probably some stuff I forgot.

The following stuff has not changed:

The stuff that shows up on your terminal when installing. Kept that exactly the same.

Stuff that needs further disccussion:
+ The CLI interface has some mentions of skipping tool dependencies and installing resolver dependencies. Skipping tool dependencies and installing resolver and repository dependencies are now the default, so these options do not make sense. They are ignored in def main but kept for backwards compatibility. (I have broken every one's scripts already once... )

I hope it is much easier to work now on shed_tools. Should also be a lot easier to write a testing framework now using pytest #80 .

Hope this somewhat redeems myself for not being at the galaxy hackathon.

This is needed when running `shed_tools test` without an explicit `--test_json` argument.

Otherwise uninstalled repos would be listed, leading to `Skipped` when you actually want to install them.

This has been deprecated in python 3.

Avoids mixups during tuple unpacking (wasn't broken here, but briefly suspected it).

if all tools are skipped or in error. Skipped tools are still tested if `--test_exisiting` is passed.

mvdbeek

This is nice, I like it in principle. I'm ambivalent on the name change of install_repositories to install_tools, because we only install repositories, and I'm afraid that distinction is lost with these changes. Which is sort-of OK, because nothing here actually deals with tools, and we were not very consistent from the start. We will break planemo though, but it's easy to update.

mvdbeek · 2018-08-05T09:40:35Z

ephemeris/get_tool_list_from_galaxy.py

@@ -176,6 +228,10 @@ def _parser():
    parser.add_argument("--get_data_managers",
                        action="store_true",
                        help="Include the data managers in the tool list. Requires login details")
+    parser.add_argument("--get_all_tools",


Getting data managers technically doesn't need an admin key, but regular users can't see the data managers, giving the wrong impression that no data managers are installed. Maybe change line 230 to Requires admin login

mvdbeek · 2018-08-05T09:44:54Z

ephemeris/shed_tools.py

+
+def log_repository_install_error(repository, start, msg, log):
+    """
+    Log failed tool installations. Return a dictionary wiyh information


s/wiyh/with

mvdbeek · 2018-08-05T09:45:30Z

ephemeris/shed_tools.py

+                repository_list.append(complete_repo)
+            except (LookupError, KeyError) as e:
+                if log:
+                    log_repository_install_error(repository, start, e.message, log)


s/e.message/str(message)/

mvdbeek · 2018-08-05T09:46:53Z

ephemeris/shed_tools.py

+                    log.debug("\tRepository %s already installed (at revision %s)" %
+                              (repository['name'], repository['changeset_revision']))
+                return "skipped"
+            elif "504" in e.message:


s/e.message/str(message)/

mvdbeek · 2018-08-05T11:16:17Z

ephemeris/shed_tools.py

+        if args.test_existing:
+            to_be_tested_tools.extend(install_results.skipped_repositories)
+
+        install_tool_manager.test_tools(


I think we should wrap this in if to_be_tested_tools:, because otherwise we start testing all tools on the instance, as you fall back to self.installed_tools() if the tool list is empty. Think if all tools were skipped but you didn't set --test_exisiting. Or if all repos errored you start testing all tools (irrespective of what you had in your tool_list.yml), that is counterintuitive.

mvdbeek · 2018-08-05T12:03:54Z

I think I have implemented all those changes in rhpvorderman#3

Rhpvorderman refactor shedtools

rhpvorderman · 2018-08-06T06:24:49Z

Thanks for all the changes @mvdbeek. Especially for fixing the error of get-tool-list listing deleted tools. I ran into this error, but did not really understand why it was happening, and then forget about it. Thanks for all the effort on this pull request!

This is nice, I like it in principle. I'm ambivalent on the name change of install_repositories to install_tools, because we only install repositories, and I'm afraid that distinction is lost with these changes. Which is sort-of OK, because nothing here actually deals with tools, and we were not very consistent from the start.

Yes this is quite ambivalent trough ephemeris. get-tool-list also gives you a list of repositories. So making this consistent across ephemeris would mean we have to use get-repository-list. Internally it can be made consistent though, to avoid confusion.

Since we are going to break planemo anyway, I better do that now to avoid breaking it twice. Commits coming up.

rhpvorderman · 2018-08-06T07:03:53Z

Renamed all the tools that are actually repos to repos.

I kept the test_tools method, since it actually tests tools. It tests all the tools in a certain list of repos. So kept that the same. I hope it is not too confusing.

Also kept the user interface the same. I can rename everything in the documentation to repositories properly, and also change the information in the parser. But I do not think that is something we should do in this pull request.

mvdbeek · 2018-08-06T07:38:47Z

.idea/vcs.xml

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8"?>


We'll want to put this file into .gitignore, right ?

Whoops. Did put this in global .gitignore. But apparently got committed anyway.

fixed now by recommitting changes

rhpvorderman added 4 commits July 18, 2018 09:26

rename shed_tools

01ddcce

start from scratch

45c40e1

include docstring

c2e486d

start with general layout for class

7783b7f

galaxyproject deleted a comment Jul 18, 2018

rhpvorderman added 7 commits July 18, 2018 09:51

move args parser elsewhere

347efc3

rationalize main function

d6c8fde

add log stuff to keep backwards compatibility

9e50cf7

start implementing install tools method

f089f11

add functionality to not flatten revisions

045be02

add code to checkout if something is already installed

8cabf79

add flatten repo info method

687122d

galaxyproject deleted a comment Jul 18, 2018

rhpvorderman added 2 commits July 18, 2018 13:41

optimize the same repository

596f4c1

wip

4dbaa19

galaxyproject deleted a comment Jul 18, 2018

rhpvorderman added 3 commits July 18, 2018 15:31

wip

1d57d66

further refactoring

6c20ba6

rewrite wait for install function

5b53916

galaxyproject deleted a comment Jul 18, 2018

fix documentation stuff

e1a3053

galaxyproject deleted a comment Jul 23, 2018

fix conflicting true false storage in parser

4adbb3f

galaxyproject deleted a comment Jul 23, 2018

fix linting issue

63bf514

galaxyproject deleted a comment Jul 23, 2018

mvdbeek added 8 commits August 5, 2018 12:13

Fix extensions typo

7a6dd40

Fix test_json default argument

b6ea9e1

This is needed when running `shed_tools test` without an explicit `--test_json` argument.

Only list repositories that are installed

7bc28cf

Otherwise uninstalled repos would be listed, leading to `Skipped` when you actually want to install them.

Listing data managers requires API access

57ac315

Avoid message attr when passing exception msg

16ede2a

This has been deprecated in python 3.

Use named attributes of namedtuple

a43bcdb

Avoids mixups during tuple unpacking (wasn't broken here, but briefly suspected it).

Fix typo

ed0ba61

Avoid testing all installed tools

cd46a4f

if all tools are skipped or in error. Skipped tools are still tested if `--test_exisiting` is passed.

mvdbeek requested changes Aug 5, 2018

View reviewed changes

Merge pull request #3 from mvdbeek/rhpvorderman-refactor_shedtools

c7bcbb5

Rhpvorderman refactor shedtools

galaxyproject deleted a comment Aug 6, 2018

mvdbeek reviewed Aug 6, 2018

View reviewed changes

rename ambigious tools to repos where applicable. Keep UI the same

c652146

rhpvorderman force-pushed the refactor_shedtools branch from 8690273 to c652146 Compare August 6, 2018 07:44

galaxyproject deleted a comment Aug 6, 2018

mvdbeek approved these changes Aug 6, 2018

View reviewed changes

rhpvorderman merged commit c082112 into galaxyproject:master Aug 6, 2018

rhpvorderman deleted the refactor_shedtools branch August 6, 2018 10:02

This was referenced Aug 6, 2018

Project: refactor shed-tools #91

Closed

shed-tool install issue with "changeset_revision" #82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Done: Refactor shedtools #104

Done: Refactor shedtools #104

rhpvorderman commented Jul 18, 2018 •

edited

mvdbeek left a comment

mvdbeek Aug 5, 2018

mvdbeek Aug 5, 2018

mvdbeek Aug 5, 2018

mvdbeek Aug 5, 2018

mvdbeek Aug 5, 2018

mvdbeek commented Aug 5, 2018

rhpvorderman commented Aug 6, 2018

rhpvorderman commented Aug 6, 2018

mvdbeek Aug 6, 2018

rhpvorderman Aug 6, 2018

rhpvorderman Aug 6, 2018

Done: Refactor shedtools #104

Done: Refactor shedtools #104

Conversation

rhpvorderman commented Jul 18, 2018 • edited

mvdbeek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek commented Aug 5, 2018

rhpvorderman commented Aug 6, 2018

rhpvorderman commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhpvorderman commented Jul 18, 2018 •

edited