Bowtie2 precompiled dependency not executable after toolshed install

[lparsons]

After installing the latest bowtie2 wrapper from the toolshed, I get a "permission denied" error when trying to run the tool. I believe this is due to the fact that the bowtie2 executable does not have the executable bit set:

-rw-r--r-- 1 galaxy galaxy 17969 Dec 16 16:30 tool-dependencies/bowtie2/2.2.4/devteam/package_bowtie_2_2_4/2b25b6e8d108/bowtie2

[dannon]

My first guess is that this is .zip not preserving permissions (.tar does). We'll repackage and it should fix things right up, thanks for pointing this out. We'll update this issue once the new package is available.

[martenson]

We re-packed the binaries and updated the package. Thanks for reporting @lparsons
24a7d71

[lparsons]

Excellent, thanks @martenson, I'll test again once the toolshed has been updated.

[lparsons]

I see that the toolshed repo package_bowtie_2_2_4 has been updated to revision 172979b6bf77, however, the repo bowtie2 still requires the older revision 2b25b6e8d108 of package_bowtie_2_2_4 so there are still some issues.

[bgruening]

@lparsons packages don't have versions anymore. If you depend on a package you always get the latest version. This shouldn't be a problem, imho.

[lparsons]

@bgruening I believe the problem is revision and not version. In my test instance of Galaxy (at latest_2014.10.06) the installed bowtie2 package shows as missing dependencies even though the latest revision 172979b6bf77 of package_bowtie_2_2_4 has been installed. I believe this is due to the fact that the toolshed inserts a revision into tool_dependencies.xml when a package is updated (if none are there in the file) and currently, the tool_dependencies.xml for bowtie2 requires revision 2b25b6e8d108 of package_bowtie_2_2_4.

[martenson]

@lparsons What I do is uninstall and install again on bowtie2 wrapper.

[bgruening]

This shouldn’t be needed, you can reinstall the dependency on package_bowtie. This should fix it hopefully.

[martenson]

@bgruening that did not work for me on Test

[lparsons]

I'm not sure if this is an issue just for my test instance (which has lots of clutter due to this type of issue) or if it would affect instances that do not have previous revisions of package_bowtie_2_2_4 installed.

That being said, I uninstalled both revisions of package_bowtie_2_2_4 and the latest version of bowtie2. When I attempt to reinstall bowtie2 Galaxy indicates three repository dependencies: package_samtools_0_1_1_8: 171cd8bc208d, package_bowtie_2_2_4: 2b25b6e8d108, and package_bowtie_2_2_4: 172979b6bf77. When I actually performed the install, it only shows a status for package_botwie_2_2_4: 172979b6bf77 and bowtie2. The bowtie2 package now shows as Installed, missing repository dependencies. It claims to be missing package_bowtie_2_2_4: 2b25b6e8d108 which shows up as an uninstalled repository on my instance.

After a restart and actually running Bowtie2, I get a "command not found" error. Looking at the logs, Galaxy has used the path: PACKAGE_BASE=/data/galaxy-dev/galaxy-dev/tool-dependencies/bowtie2/2.2.4/devteam/bowtie2/c1ec08cb34f9 which actually only contains an env.sh file that points to a (non-existant) /data/galaxy-dev/galaxy-dev/tool-dependencies/bowtie2/2.2.4/devteam/package_bowtie_2_2_4/2b25b6e8d108/env.sh.

All in all, quite the mess. It there any recommended way of truly removing installed toolshed repositories from a Galaxy instance (purging, so to speak). There are quite a few duplicate entries like this in my test instance now.

[davebx]

@lparsons have you also tried installing the uninstalled package_bowtie_2_2_4?

[lparsons]

@InitHello The uninstalled package_bowtie_2_2_4 shows up as "updates available". The initial reinstallation attempt generated an error message on the admin webpage (claimed it failed). The paster.log shows Skipping installation of revision 172979b6bf77 of repository 'package_bowtie_2_2_4' because it was installed with the (possibly updated) revision 172979b6bf77 and its current installation status is 'Installed'. The status of package_bowtie_2_2_4: 2b25b6e8d108 remains New.

[lparsons]

I believe the underlying issue is that Galaxy can get into a state where there are multiple records for the same toolshed repository version. I don't think this should happen, but I know of no simple way of cleaning it up from this state since the records are not removed from the database.

[martenson]

I would try clean uninstall and clean install. I suspect that the 'install updates' caused some issues I encountered on Test Galaxy yesterday.

[lparsons]

@martenson I'm not quite sure what you mean by 'clean uninstall' and 'clean install'. What sort of a process can I use to do that?

[martenson]

@lparsons Galaxy gives you an option to deactivate or uninstall any repository. I meant the uninstall.

[lparsons]

@martenson Oh, I see, I've done that, but that simply deletes files on disk (sometimes). There is no way to remove what I think are "duplicate" entries in whatever table(s) Galaxy is using to track what repositories are installed. Thus, there is no real way to do a "clean install" since it's always viewed by Galaxy as "reinstalling" (except when it isn't and a duplicate records gets created, I've no idea how that happens).

[jmchilton]

@lparsons MySQL or postgres?

[lparsons]

@jmchilton Postgres 8.4.20, CentOS release 5.10 (Final)

[jmchilton]

:( I would think the purpose of the database in to enforce integrity that would be hard to do with the file system alone. Erg...

[bgruening]

@lparsons have you tried the 'repair repository' function?

[lparsons]

I agree @jmchilton.

@bgruening, yes, I have tried that but it really only seems to automate the uninstall reinstall loop.

I believe we have two main tasks at this point:

1. Determine if it is "safe" to install the updated bowtie2 repository in existing Galaxy instances (specifically, my current production instance).
2. Start a Trello card to track the issue(s) that cause Galaxy to end up with multiple instances of the same toolshed repository version in the database.

Since the installed repository duplication bug is a long standing issue for (that has plagued me since the start of the toolshed), I would very much like to help define the problem more clearly so we can get a resolution. That being said, I'm not quite sure of the best way to do that. I think at a minimum we need a Galaxy team member to be able to duplicate the issue.

[lparsons]

So, after quite a bit of wrangling, I was able to "purge" one of the "package" repositories and then things started to work much better. However, there is still a major issue that I believe @jmchilton is working on: http://gmod.827538.n3.nabble.com/Error-when-opening-bowtie2-tool-page-tp4047540.html. Is there a Trello card or GitHub issue on this?

Also, I ran into another instance of a duplicated toolshed repositories during this (now I have two instances of the same bowtie2 toolshed repo on my test instance). Should I create a Trello card for that, or would someone else prefer to do it (who maybe has a better handle on the issue and how to word things)?