What is your recommandation for external plugins?
Where should we store the ZIP file? Is Sonatype the "best" place for it or will you open download.elasticsearch.org for plugin developers ?

Hi Shay, did you have a chance to look at elasticsearch-apps by Jörg? I think this is pretty good approach.

@kimchy thanks for your kind feedback about you want to evaluate https://github.com/jprante/elasticsearch-apps
Will 3rd party developers have to register on elasticsearch.org or will they be able to offer additional download locations?

@dadoonet I agree that Maven repo addressing would be a viable solution for many plugins right now, it should be non-obtrusive to those who do not want to use Maven though. Beside Maven central / Sonatype, adding private repos should also be possible, to convince organizations that ES can rely on a self-controlled plugin ecosystem.

@lukas-vlcek thanks for the hint to https://github.com/jprante/elasticsearch-apps :) I have opened a showcase of an ES 0.20.1 patched version, using a tool bin/app as a replacement for bin/plugin to show how a Maven repo dependency mechanism can be used from within an ES node startup.

Just a note because github is now excluded: for small "github blobs" (around 1 MB or so?), some plugin zips could still be downloaded from within the repo area of github, via a HTTP GET from https://raw.github.com so I think it would also be nice to still include github, for example see #2477

I find it very useful, as hinted to me by @kimchy, to offer the plugin mechanism also to non-Java folks and to make the plugins more polyglot in the future.

By re-using existing repo infrastructure out there, the core team won't have to monitor download.elasticsearch.org for availability and packaging policies - less janitor duties means more focusing on Elasticsearch core development :)

There are the ruby gems and the python easy_install repos out there. Of course, perl has CPAN (but does not run within a JVM).

Maven 3 is getting more and more polyglot and is no longer Java-centric.

Anyway, there is still work to do for real polyglot solutions. Smarter people than me were already busy with this. One approach showed how to package JRuby gems in a jar for Maven

http://blog.nicksieger.com/articles/2009/01/10/jruby-1-1-6-gems-in-a-jar/

and another one is for standalone Jython scripts

http://mavenjython.sourceforge.net/

There is also a method for translating the POM into other languages - the Maven-specific polyglot approach

https://github.com/tobrien/polyglot-maven

I think it is not so relevant for ES plugin repo substitution to the missing github download right now, but it shows the power and flexibility of the Maven 3 core.

So I am optimistic there will be some usefulness in an unobtrusive Maven repo distribution method that may be valuable also for ES polyglot plugins.

I've got my answer in the 0.20.2 release note:

Another change includes how plugins are downloaded. Now, “elasticsearch” plugins are downloaded automatically from our new download.elasticsearch.org service, but it can also download plugins from maven directly. Site plugins can stil be downloaded from github as “repositories”. In the future, we will also allow users to upload their own plugins to download.elasticsearch.org if they wish to.

Just to offer a different perspective (non-Java developer, non-plugin developer): regardless of the support for "common plugin locations", you can install a plugin by using it's fully qualified URL, right? This is what eg. the Chef cookbook supports, allowing to install public or private plugins.

Heya,

First, a fully qualified URL still works when installing plugins, as @karmi mentioned, so you can always host it wherever you wish to.

Regarding external plugins, except for the full qualified URL, the plugin mechanism will also try and install it from maven. So you can simply publish it to sonatype (or maven central through it), and it will download it from there. I guess in any case as a plugin developer you would like to provide the plugin on maven repo, so the current solution would work nicely there as well.

Last, we might also end up allowing for people to upload plugins to download.elasticsearch.org. We will see. Its not a big effort on our end, assuming that maven is not enough, we can definitely do that.

Lets see how this solution evolves, and see what is left needed, and we can tackle it then. Its very simple now, but covers a lot of cases, I tend to like these type of solutions :)

With this solution, I think I have to clean up the artifact group ids of my plugin jars to something like org.xbib.elasticsearch.plugins and I also have publish full dropbox URLs or something in my plugin READMEs per version, because I'm not using the default repo at Maven central or Sonatype OSSRH.

Note, looking up a ZIP under http://search.maven.org/ is not using Maven in the intended way, it uses just another central download service add-on, like github before. That's not the way a Maven repo is supposed to work. The typical use of Maven is resolving dependencies, and having the opportunity for each user to add a lot of other public and private Maven repos too, with different scopes and terms of use.

I am all for allowing uploads to download.elasticsearch.org - having a bunch of anywhere hosted plugins will make them every now and then unreachable. We could even build a nice web gui on top of that repo and make them searchable very easily (along with its documentation, all centralized)...

I once assumed plugin development and making plugins downloadable should be decentralized, but I may be wrong.

Note, each download service should state to users who is responsible for the uploaded file. The legal stuff must be taken care of, such as different licensing of plugins, and copyright infringement protection need to be sorted out so that download.elasticsearch.org can't be taken down simply by a voluntary DMCA action against a suspicious plugin. What kind of copyright law applies, is not really clear to me, I think it's mostly the law of the country of the downloader.

Another option would be an obligation for donating plugin code to download.elasticsearch.org under the Apache license with a CLA. By doing this, plugins could easily be moved into the ES core at will.

Additionally, mirrors of download.elasticsearch.org should be considered, in case of planned or unplanned outages.

I guess maintaining download.elasticsearch.org has limited resources to handle all of these things. I'm afraid the volunteering technical staff will not scale very well with the hopefully ever growing number of Elasticsearch plugins.

@jprante agreed, those are good questions you asked for regarding the download service, which we need to figure out if we allow for people to upload plugins to download.elasticsearch.org. For now though, "code plugins" can easily be downloaded from maven as well, so I don't see a problem there (plugin -install <group_id>/<artifact_id>/<version>).