Tool Developers Meeting (February 2015)

[jmchilton]

At the end of the last Tool Developers meeting we decided to make it a bi-monthly thing. I have put together a Doodle poll with some time slots. It is still a while from now so probably best to fill out the poll closer to that date but I wanted to make sure it was on the radar.

http://doodle.com/q634w4ae2nhgaf84

Once we have decided on a time slot we can post of galaxy-dev (which we should have done last time).

[hexylena]

Now that a 1/3rd of the month is over... is anyone else in @galaxyproject/iuc still interested in a tool developer meeting?

[jmchilton]

Doodle somehow seems to think February 23 at 4:00 PM penn state time (21:00 UTC?) works for everyone (spanning three continents) so I say we go with that. (I would have thought that was too late for Europe - did people see the poll in their own timezones?)

@bgruening does that work for you still?

[peterjc]

Not ideal, but I decided I could live with a 9pm local time meeting given the broad range of time zones.

If the poll had been setup to allow yellow "maybe" (rather than just red "no" and green "yes") I would have ticked that ;)

[bgruening]

@jmchilton it's ok. Someone have to bite the bullet :)
I will be in Slovenia (Galaxy workshop) so this fits nicely. Hopefully internet works this time!

[jmchilton]

Alright thanks Europeans - if @fubar2 doesn't show up though we will make the next one a lot earlier in the day :).

[hexylena]

Yeah, it's no problem for me to be up at 6-9 am (1200 to 1500 UTC) if that's easier on the europeans. Personally would've assumed that early was better than late. But 2015-02-23T21:00Z works for me.

[jmchilton]

Yeah - lets put some earlier times on the next poll (if we do another poll) - I started at 9 AM EST because I gather Marten is not a fan of getting up that early and several of the rest of us at Penn State need to get kid(s) to daycare/school in the morning - which can make things difficult.

[martenson]

I am definitely a fan of getting up early. Lets make it 3am next time!
M.
p.s. I really don't care, schedule it any time.

On Tue Feb 10 2015 at 1:03:49 PM John Chilton notifications@github.com
wrote:

Yeah - lets put some earlier times on the next poll (if we do another
poll) - I started at 9 AM EST because I gather Marten is not a fan of
getting up that early and several of the rest of us at Penn State need to
get kid(s) to daycare/school in the morning - which can make things
difficult.

—
Reply to this email directly or view it on GitHub
#23 (comment)
.

[jj-umn]

Anytime works from as well.

[jmchilton]

Pinging everyone on this thread to remind them this is happening tomorrow. We should probably put together an agenda and announce this on -dev.

[hexylena]

Agenda, yes please!

søn. 22. feb. 2015, 19.59 skrev John Chilton notifications@github.com:

Pinging everyone on this thread to remind them this is happening tomorrow.
We should probably put together an agenda and announce this on -dev.

—
Reply to this email directly or view it on GitHub
#23 (comment)
.

[bgruening]

Proposed items:

• automated testing tools-iuc / tools-devteam
• planemo developments, priorities until GCC
• Tool Shed developments, priorities until GCC
• Tool Shed 2.0, brew vs. conda
• automated tool_dependencies.xml building
• small enhancements that makes tool development easier, Trello round-up
• afaik Anton was calling for 2015 the year of tools, do we have priorities for some tools?
• IUC at GCC, we should present ourself and recruit new peoples (Poster, Talk ...)
• rating of repositories, every year we plan to enhance this and rate our repositories ...

[fubar2]

I'll be there - 8am AEST is the first sensible meeting time I've seen in ages. I hope my dressing gown will meet or exceed the minimum sartorial standards for a hangout?
:)

If there's time on the agenda, I have two potential additions:

• Expose iuc tools for testing on, um, drumroll...., test? eg https://biostar.usegalaxy.org/p/10737/ would be much easier to debug if the user could submit an error report (yes, it looks like lance parsons' htseq tool but the principle is the same!).
• steps and processes to rationalise multiple tools designed to do much the same thing - eg there are several htseq based count packages in the test and main toolsheds - how do we pick "winners" to support or do we 'let a hundred flowers blossom' ?

[jmchilton]

I was worried we would have trouble coming up with something to talk about - but it actually looks like we have more to talk about than last time. So I have put together notes in the form of slides that we can cycle through - please feel free to go in and add stuff now especially if there is something you want to talk about on any of these topics.
https://docs.google.com/presentation/d/1i9GD_FmaqZmFDAbp4VSvuAB7Sw0WsnyUr5Vdk6ks4dk/edit?usp=sharing

These notes shouldn't be construed to mean this will be an interactive lecture formatted meeting like GalaxyAdmins say - I like the informal conversation style of the last meeting - I think that is the right approach - I am hoping we can do the same thing with slides in front of us.

[peterjc]

Those slides should say 2015-02-23 right? Not 2015-02-22.

This is at 4pm / 16:00 EST or 9pm / 21:00 GMT/UTC, meaning in about 10 hours time?

[hexylena]

@peterjc 2015-02-23T21:00Z, yep, about 8 hours from now

[jmchilton]

https://plus.google.com/hangouts/_/gwutcfpyrqq3reoeu3rumb23dua

[hexylena]

Depot files proposal:

• we want need files that are essentially permanently available
• we want multiple people to host copies
• files shouldn't be able to be removed from the network
• we haven't a need for versioning (so git makes less sense)

That screams bittorrent to me. A tracker to point at a peer that the files can be retrieved from, multiple peers to ensure permanent redundancy. Thoughts? (This sounds like a "fun" project and I've been playing with DHTs lately)

[bgruening]

@erasche I like it, at our last workshop we tried to distribute docker images via bittorrent.

[hexylena]

@bgruening great to hear. I feel like if the galaxyproject and at least one or two other organisations provided some storage space (I'll volunteer the VM I use for jenkins), we could really provide a longer lasting service here for galaxy specific storage of binary blobs that are needed for reproducible research.

Extended thoughts

• would require some sort of approval system for adding files, i.e. a privileged peer who could add files after review (especially if there was no functionality to remove files from the table)
• might implement a system for letting people say "I have N GB of storage, decide what I should host" or "I'd like to host a copy of X (e.g. perl modules) because they're important to me"
• each file would be a torrent, clients would automatically fetch latest frm an RSS feed (or use their built in decision mechanism for files)

This could definitely be a possibility! Notes to self:

• https://github.com/chihaya/chihaya
• http://mktorrent.sourceforge.net/
• http://stackoverflow.com/questions/5400828/how-to-write-a-simple-bittorrent-application

[martenson]

I know someone who might be interested. Istvan Albert (who runs https://biostar.usegalaxy.org/) implemented bittorent tracker for the Biostar site (to share datasets amongst users).

edit: I am not sure about the current state of the implementation, but it is probably done without frontend yet. But we might not need one. https://www.biostars.org/p/106585/
I can get more information regarding this.

[bgruening]

ping @ialbert :)

[hexylena]

Ah, that's what I get for digging up an email address and emailing him. Thanks @bgruening :)

[bgruening]

You're welcome, as usual :)

[hexylena]

@martenson thanks for that post, too bad that BioTorrents has been dead for a few years by the looks of it.

[hexylena]

The other option is running a Tahoe-LAFS cluster amongst participating organisations, but that would require removal prevention hacked in (and I'd have to dig into how it handles things like permanent node failure and accessibility of data). I think that has some form of web front end already built (we really only need a dumb http server for the files, but if it's built in...)

[martenson]

Writing Galaxy bittorent tracker can be a nice project too. About two weeks. :)

[ialbert]

We do have a tracker in biostar in a reasonably factored out manner

https://github.com/ialbert/biostar-central/tree/tracker/biostar/apps/tracker

Our primary issue ended up with having the bittorrent ports blocked for
different providers.

It is still something we are interested in but there is also a new kid on
the block, webtorrent that promises the same thing but as WebRCTP requests
over standard HTTP ports. I think that is the future in that cannot be
blocked and large data transfers will take place with that protocols anyhow

https://github.com/feross/webtorrent

On 24 February 2015 at 11:29, Martin Cech notifications@github.com wrote:

Writing Galaxy bittorent tracker can be a nice project. :)

—
Reply to this email directly or view it on GitHub
#23 (comment)
.

[hexylena]

@ialbert Thanks for sharing that! That definitely sounds like a preferable alternative to dealing with institutions and blocked ports, though I hadn't been intending to use it for providing the file downloads.

With something like that, it's conceivable that galaxy instances installing packages could contribute to the swarm.

[hexylena]

Meeting is completed (next time I'll video/take minutes)