Winter Break Sprint

[jmchilton]

Host the Tarballs (for @erasche)

• Implement the planemo API for tool publication. (Scoped out for testing here - https://github.com/galaxyproject/planemo/blob/master/tests/shed_app.py).
• Implement the tool shed 1.0 search API - or something very similar. (Probably reuse backend technologies.) This will allow this to be queries by toolshed.tools (https://github.com/martenson/toolshed.tools).
• Implement validation of inputs:
  • Implement galaxy-lib project so TS 2.0 can reason about tools with same code as Galaxay & Planemo.
  • Tools need to have strictly increasing versioning.
  • Tools must be namespaced.
  • Only one tool may exist per tar ball.
• Shed features
  • handle download of files
  • more tests.
  • parse requirements
  • upload all of the IUC's stuff until things break
• planemo stuff
  • upload sha256s
  • upload gpg asc files
  • support shed json objects cast repo_id as string for shed compat galaxyproject/planemo#398

Later

• Support if possible CWL tools and the CWL-service interface (https://github.com/common-workflow-language/cwltool-service). This is highly alpha.

Install the Tarballs (Backend) (for @jmchilton)

• Create an API within a Galaxy-style management app (that can be run standalone or as just more controllers in Galaxy) for managing Tool Shed 2.0 installs.
  • Download tarball, verify contains a valid tool.
    • Extend tool auto-finding to look for any valid tool using the galaxy parser module.
  • Extract to <target_dir>/<id>/<version>/<revision=0>/source/.
  • Create <target_dir>/<id>/<version>/<revision=0>/install_info.json.
  • Implement JSON/YAML tool confs. ToolBox Overhaul galaxyproject/galaxy#1398
  • Update tool conf <target_dir>/shed_conf.json.
  • Implement toolbox reloading. ToolBox Overhaul galaxyproject/galaxy#1398
  • Watch all tool_confs and reload on change. ToolBox Overhaul galaxyproject/galaxy#1398
  • Add a watchdog hard dependency to Galaxy (Add watchdog required dependency for Galaxy galaxyproject/starforge#51). (Watchdog only watches directories, more performant to just monitor files I think.)
• Implement a conda plugin for dependency resolver framework used by galaxy/planemo/pulsar. Implement a conda dependency resolver. galaxyproject/galaxy#1345
• Implement a Galaxy-style plugin framework for "managing" a list of Galaxy-style requirements (not just resolving).
  • List available packages and versions
  • Allow query of upstream/source versions and packages.
  • Allow installation of dependencies.
  • Allow installation of dependencies (async).
  • Allow uninstallation of dependencies.
  • Allow uninstallation of dependencies (async).
  • Allow debugging of installation issues.
• Implement a conda backend for the above framework. (Bonus points for platform-brew).
• Implement an API within Galaxy-style management app for communicating for managing requirements framework.

Install the Tarballs (Frontend) (for @martenson, @erasche, and @jmchilton )

• Extend https://github.com/martenson/toolshed.tools to be a full client to the Tool Shed 2.0 - make this the GUI for management app described abov. Adding options for installing, etc.... (See comment below for potential breakdown.)
  • Search to find repositories containing tools. (Talking to the Tool Shed 1.0/2.0 search API).
  • Install a tool. (Talking to Galaxy-style management API).
  • Allow installing many tools at once/in batch.
  • Handling suites.
  • Allow browsing detailed metadata, tool previews, etc.... or individual search results. (maybe?)
  • Allow browsing through categories to find tools.
• Implement a GUI on top of the API for managing requirements described above.
• Implement a GUI tying together installation of tools and their requirements.

Application for Publishing the Tarballs. (Stretch goal)

• Create a backend database schema to map github repos to tool shed 1.0/2.0 repos using planemo. Implement Github/GitURL -> Shed persistent mapping backend. galaxyproject/planemo#177
• Establish a RESTful API for the above mapping that be used standalone, in TS 1.0 or 2.0.
• Create a UI for targetting above RESTful API.
• Decide if it would be better to host a standalone Planemo service or bake it into the Tool Shed 2.0 using the above API and backend. @erasche thinks baked in would be great and is happy to add that to his TS2.0 UI

[jmchilton]

Ping @erasche - what do you think? Some of the galaxy specific stuff like validating versions and stuff - you can just stub out something in planemo and I'd be happy to fill it in.

[hexylena]

@jmchilton I think this is awesome, thanks a million for putting this together. I'll help out on the Galaxy side however I can (given your much longer list :/) but as usual I'm less familiar with the code.

"Extend https://github.com/martenson/toolshed.tools to be a full client to the Tool Shed 2.0, back into this management app and add options for installing, etc...."

do you mean just a JS WebApp for browsing the toolshed? If so I'd be happy to throw an angularJs toolshed browser/editor onto my list, since this repo basically contains that already.

[jmchilton]

@erasche I mean I wanted to use https://github.com/martenson/toolshed.tools/ as the basis for the Galaxy (and here I mean Galaxy-style management app that may or may not run as part of Galaxy) client to the tool shed. Here I guess in order of importance what this JS should do it:

• Search to find repositories containing tools. (Talking to the Tool Shed 1.0/2.0 search API).
• Install a tool. (Talking to Galaxy-style management API).
• Cart-style interface to install many tools.
• Handling suites.
• Allow browsing detailed metadata, tool previews, etc.... or individual search results. (maybe?)
• Allow browsing through categories to find tools.

If you want to take over this piece and make it angular that is fine by me. The existing Backbone stuff from @martenson - is much more stylish than anything I could do and uses technologies very compatible with Galaxy (maybe just Backbone and Font Awesome).

[martenson]

If you don't mind I will jump in and work on the search interface with you guys.

[hexylena]

Yeah, I'd rather not force yet-another-js lib on people... time to leqrn
backbone! :-) what's a sprint without leqrning an entire new way to.write
JS
15. des. 2015 10.26 p.m. skrev "John Chilton" notifications@github.com:

@erasche https://github.com/erasche I mean I wanted to use
https://github.com/martenson/toolshed.tools/ as the basis for the Galaxy
(and here I mean Galaxy-style management app that may or may not run as
part of Galaxy) client to the tool shed. Here I guess in order of
importance what this JS should do it:

• Search to find repositories containing tools. (Talking to the Tool
  Shed 1.0/2.0 search API).
• Install a tool. (Talking to Galaxy-style management API).
• Cart-style interface to install many tools.
• Handling suites.
• Allow browsing detailed metadata, tool previews, etc.... or
  individual search results. (maybe?)
• Allow browsing through categories to find tools.

If you want to take over this piece and make it angular that is fine by
me. The existing Backbone stuff from @martenson
https://github.com/martenson - is much more stylish than anything I
could do and uses technologies very compatible with Galaxy (maybe just
Backbone and Font Awesome).

—
Reply to this email directly or view it on GitHub
#1 (comment).

[jmchilton]

@martenson

• Is it realistic to generalize that code so it can be used standalone still but also within a Galaxy-style management application?
• How far do you want to extend the search interface? Do you want to do the cart-style checkouts? Do you want to do the browsing of tool shed by categories?

I'll go ahead then and pull all the GUI stuff out of my "section" since I don't want to do it anyway (I like to stop my development at the API endpoint) - and then you guys can call dibs on the pieces.

[jmchilton]

Added a new section to this sprint - "Application for Publishing the Tarballs. (Stretch goal)" to further my planemo agenda. If this piece is designed well I think we could reuse it for either the TS 1.0, TS 2.0, or just as a stand-alone thing.

[martenson]

• yes
• I do not understand the 'how far to extend search interface' question
• I am quite opposed having a cart for tools; I have few ideas how to make it better I think
• browsing categories: yes

[jmchilton]

@martenson In response to point 2, it sounds like you want to take over like the full GUI to the tool shed on the Galaxy side. This is perfect, thanks!

[martenson]

Yes, that is my goal, so many 'distractions' though. Winter break is a good time to focus.

I hope it will serve for TS 1.0 and 2.0 and whatever other versions we come up with.

[jmchilton]

Not TS 4.0, that jerk 🐦 👀 can do his own UI.

[hexylena]

@martenson

I am quite opposed having a cart for tools; I have few ideas how to make it better I think

Oooh! exciting. Thanks for joining us :)

[jmchilton]

Yes, I'll update the language to say install multiple tools easily - whatever that looks like.

[hexylena]

https://shed.hx42.org/

Getting closer to usability. Code is pulled and process restarted hourly because I was too tired to do it properly while we're in the "super secret alpha code" phase.

The 404 at that page is intentional, you can use the django error pages to get hints as to what URLs are available.

[martenson]

Sorry for the lack of contributions @erasche & @jmchilton. Xmass got more lazy and have less time than I estimated.

[hexylena]

Apologies for delays. PI's course started--all hell broke loose. Then started looking for a house in town and that was another level of cluster-fuckiness.

Will wrap this up soon/during PSU visit. I want to get rid of old cruft once and for all :)