New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugins Installer: Allow to download plugins from download.elasticsearch.org #2507

Closed
kimchy opened this Issue Dec 26, 2012 · 10 comments

Comments

Projects
None yet
6 participants
@kimchy
Member

kimchy commented Dec 26, 2012

The plugin script now will automatically download plugins from download.elasticsearch.org instead of github. It will also try maven (both central and sonatype) if it fails.

Effectively, this change removes the ability to install plugins from the downloads section in github, since its no longer supported by github. Site plugins can still be installed by downloading the "repo".

Direct installation is still supported, using the bin/plugin -url file://path/to/plugin -name plugin-name.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Dec 27, 2012

Member

What is your recommandation for external plugins?
Where should we store the ZIP file? Is Sonatype the "best" place for it or will you open download.elasticsearch.org for plugin developers ?

Member

dadoonet commented Dec 27, 2012

What is your recommandation for external plugins?
Where should we store the ZIP file? Is Sonatype the "best" place for it or will you open download.elasticsearch.org for plugin developers ?

@lukas-vlcek

This comment has been minimized.

Show comment
Hide comment
@lukas-vlcek

lukas-vlcek Dec 27, 2012

Contributor

Hi Shay, did you have a chance to look at elasticsearch-apps by Jörg? I think this is pretty good approach.

Contributor

lukas-vlcek commented Dec 27, 2012

Hi Shay, did you have a chance to look at elasticsearch-apps by Jörg? I think this is pretty good approach.

@jprante

This comment has been minimized.

Show comment
Hide comment
@jprante

jprante Dec 27, 2012

Contributor

@kimchy thanks for your kind feedback about you want to evaluate https://github.com/jprante/elasticsearch-apps
Will 3rd party developers have to register on elasticsearch.org or will they be able to offer additional download locations?

@dadoonet I agree that Maven repo addressing would be a viable solution for many plugins right now, it should be non-obtrusive to those who do not want to use Maven though. Beside Maven central / Sonatype, adding private repos should also be possible, to convince organizations that ES can rely on a self-controlled plugin ecosystem.

@lukas-vlcek thanks for the hint to https://github.com/jprante/elasticsearch-apps :) I have opened a showcase of an ES 0.20.1 patched version, using a tool bin/app as a replacement for bin/plugin to show how a Maven repo dependency mechanism can be used from within an ES node startup.

Just a note because github is now excluded: for small "github blobs" (around 1 MB or so?), some plugin zips could still be downloaded from within the repo area of github, via a HTTP GET from https://raw.github.com so I think it would also be nice to still include github, for example see #2477

I find it very useful, as hinted to me by @kimchy, to offer the plugin mechanism also to non-Java folks and to make the plugins more polyglot in the future.

By re-using existing repo infrastructure out there, the core team won't have to monitor download.elasticsearch.org for availability and packaging policies - less janitor duties means more focusing on Elasticsearch core development :)

There are the ruby gems and the python easy_install repos out there. Of course, perl has CPAN (but does not run within a JVM).

Maven 3 is getting more and more polyglot and is no longer Java-centric.

Anyway, there is still work to do for real polyglot solutions. Smarter people than me were already busy with this. One approach showed how to package JRuby gems in a jar for Maven

http://blog.nicksieger.com/articles/2009/01/10/jruby-1-1-6-gems-in-a-jar/

and another one is for standalone Jython scripts

http://mavenjython.sourceforge.net/

There is also a method for translating the POM into other languages - the Maven-specific polyglot approach

https://github.com/tobrien/polyglot-maven

I think it is not so relevant for ES plugin repo substitution to the missing github download right now, but it shows the power and flexibility of the Maven 3 core.

So I am optimistic there will be some usefulness in an unobtrusive Maven repo distribution method that may be valuable also for ES polyglot plugins.

Contributor

jprante commented Dec 27, 2012

@kimchy thanks for your kind feedback about you want to evaluate https://github.com/jprante/elasticsearch-apps
Will 3rd party developers have to register on elasticsearch.org or will they be able to offer additional download locations?

@dadoonet I agree that Maven repo addressing would be a viable solution for many plugins right now, it should be non-obtrusive to those who do not want to use Maven though. Beside Maven central / Sonatype, adding private repos should also be possible, to convince organizations that ES can rely on a self-controlled plugin ecosystem.

@lukas-vlcek thanks for the hint to https://github.com/jprante/elasticsearch-apps :) I have opened a showcase of an ES 0.20.1 patched version, using a tool bin/app as a replacement for bin/plugin to show how a Maven repo dependency mechanism can be used from within an ES node startup.

Just a note because github is now excluded: for small "github blobs" (around 1 MB or so?), some plugin zips could still be downloaded from within the repo area of github, via a HTTP GET from https://raw.github.com so I think it would also be nice to still include github, for example see #2477

I find it very useful, as hinted to me by @kimchy, to offer the plugin mechanism also to non-Java folks and to make the plugins more polyglot in the future.

By re-using existing repo infrastructure out there, the core team won't have to monitor download.elasticsearch.org for availability and packaging policies - less janitor duties means more focusing on Elasticsearch core development :)

There are the ruby gems and the python easy_install repos out there. Of course, perl has CPAN (but does not run within a JVM).

Maven 3 is getting more and more polyglot and is no longer Java-centric.

Anyway, there is still work to do for real polyglot solutions. Smarter people than me were already busy with this. One approach showed how to package JRuby gems in a jar for Maven

http://blog.nicksieger.com/articles/2009/01/10/jruby-1-1-6-gems-in-a-jar/

and another one is for standalone Jython scripts

http://mavenjython.sourceforge.net/

There is also a method for translating the POM into other languages - the Maven-specific polyglot approach

https://github.com/tobrien/polyglot-maven

I think it is not so relevant for ES plugin repo substitution to the missing github download right now, but it shows the power and flexibility of the Maven 3 core.

So I am optimistic there will be some usefulness in an unobtrusive Maven repo distribution method that may be valuable also for ES polyglot plugins.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Dec 28, 2012

Member

I've got my answer in the 0.20.2 release note:

Another change includes how plugins are downloaded. Now, “elasticsearch” plugins are downloaded automatically from our new download.elasticsearch.org service, but it can also download plugins from maven directly. Site plugins can stil be downloaded from github as “repositories”. In the future, we will also allow users to upload their own plugins to download.elasticsearch.org if they wish to.

Member

dadoonet commented Dec 28, 2012

I've got my answer in the 0.20.2 release note:

Another change includes how plugins are downloaded. Now, “elasticsearch” plugins are downloaded automatically from our new download.elasticsearch.org service, but it can also download plugins from maven directly. Site plugins can stil be downloaded from github as “repositories”. In the future, we will also allow users to upload their own plugins to download.elasticsearch.org if they wish to.

@karmi

This comment has been minimized.

Show comment
Hide comment
@karmi

karmi Dec 29, 2012

Member

Just to offer a different perspective (non-Java developer, non-plugin developer): regardless of the support for "common plugin locations", you can install a plugin by using it's fully qualified URL, right? This is what eg. the Chef cookbook supports, allowing to install public or private plugins.

Member

karmi commented Dec 29, 2012

Just to offer a different perspective (non-Java developer, non-plugin developer): regardless of the support for "common plugin locations", you can install a plugin by using it's fully qualified URL, right? This is what eg. the Chef cookbook supports, allowing to install public or private plugins.

@kimchy

This comment has been minimized.

Show comment
Hide comment
@kimchy

kimchy Jan 6, 2013

Member

Heya,

First, a fully qualified URL still works when installing plugins, as @karmi mentioned, so you can always host it wherever you wish to.

Regarding external plugins, except for the full qualified URL, the plugin mechanism will also try and install it from maven. So you can simply publish it to sonatype (or maven central through it), and it will download it from there. I guess in any case as a plugin developer you would like to provide the plugin on maven repo, so the current solution would work nicely there as well.

Last, we might also end up allowing for people to upload plugins to download.elasticsearch.org. We will see. Its not a big effort on our end, assuming that maven is not enough, we can definitely do that.

Lets see how this solution evolves, and see what is left needed, and we can tackle it then. Its very simple now, but covers a lot of cases, I tend to like these type of solutions :)

Member

kimchy commented Jan 6, 2013

Heya,

First, a fully qualified URL still works when installing plugins, as @karmi mentioned, so you can always host it wherever you wish to.

Regarding external plugins, except for the full qualified URL, the plugin mechanism will also try and install it from maven. So you can simply publish it to sonatype (or maven central through it), and it will download it from there. I guess in any case as a plugin developer you would like to provide the plugin on maven repo, so the current solution would work nicely there as well.

Last, we might also end up allowing for people to upload plugins to download.elasticsearch.org. We will see. Its not a big effort on our end, assuming that maven is not enough, we can definitely do that.

Lets see how this solution evolves, and see what is left needed, and we can tackle it then. Its very simple now, but covers a lot of cases, I tend to like these type of solutions :)

@jprante

This comment has been minimized.

Show comment
Hide comment
@jprante

jprante Jan 7, 2013

Contributor

With this solution, I think I have to clean up the artifact group ids of my plugin jars to something like org.xbib.elasticsearch.plugins and I also have publish full dropbox URLs or something in my plugin READMEs per version, because I'm not using the default repo at Maven central or Sonatype OSSRH.

Note, looking up a ZIP under http://search.maven.org/ is not using Maven in the intended way, it uses just another central download service add-on, like github before. That's not the way a Maven repo is supposed to work. The typical use of Maven is resolving dependencies, and having the opportunity for each user to add a lot of other public and private Maven repos too, with different scopes and terms of use.

Contributor

jprante commented Jan 7, 2013

With this solution, I think I have to clean up the artifact group ids of my plugin jars to something like org.xbib.elasticsearch.plugins and I also have publish full dropbox URLs or something in my plugin READMEs per version, because I'm not using the default repo at Maven central or Sonatype OSSRH.

Note, looking up a ZIP under http://search.maven.org/ is not using Maven in the intended way, it uses just another central download service add-on, like github before. That's not the way a Maven repo is supposed to work. The typical use of Maven is resolving dependencies, and having the opportunity for each user to add a lot of other public and private Maven repos too, with different scopes and terms of use.

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jan 7, 2013

Member

I am all for allowing uploads to download.elasticsearch.org - having a bunch of anywhere hosted plugins will make them every now and then unreachable. We could even build a nice web gui on top of that repo and make them searchable very easily (along with its documentation, all centralized)...

Member

spinscale commented Jan 7, 2013

I am all for allowing uploads to download.elasticsearch.org - having a bunch of anywhere hosted plugins will make them every now and then unreachable. We could even build a nice web gui on top of that repo and make them searchable very easily (along with its documentation, all centralized)...

@jprante

This comment has been minimized.

Show comment
Hide comment
@jprante

jprante Jan 7, 2013

Contributor

I once assumed plugin development and making plugins downloadable should be decentralized, but I may be wrong.

Note, each download service should state to users who is responsible for the uploaded file. The legal stuff must be taken care of, such as different licensing of plugins, and copyright infringement protection need to be sorted out so that download.elasticsearch.org can't be taken down simply by a voluntary DMCA action against a suspicious plugin. What kind of copyright law applies, is not really clear to me, I think it's mostly the law of the country of the downloader.

Another option would be an obligation for donating plugin code to download.elasticsearch.org under the Apache license with a CLA. By doing this, plugins could easily be moved into the ES core at will.

Additionally, mirrors of download.elasticsearch.org should be considered, in case of planned or unplanned outages.

I guess maintaining download.elasticsearch.org has limited resources to handle all of these things. I'm afraid the volunteering technical staff will not scale very well with the hopefully ever growing number of Elasticsearch plugins.

Contributor

jprante commented Jan 7, 2013

I once assumed plugin development and making plugins downloadable should be decentralized, but I may be wrong.

Note, each download service should state to users who is responsible for the uploaded file. The legal stuff must be taken care of, such as different licensing of plugins, and copyright infringement protection need to be sorted out so that download.elasticsearch.org can't be taken down simply by a voluntary DMCA action against a suspicious plugin. What kind of copyright law applies, is not really clear to me, I think it's mostly the law of the country of the downloader.

Another option would be an obligation for donating plugin code to download.elasticsearch.org under the Apache license with a CLA. By doing this, plugins could easily be moved into the ES core at will.

Additionally, mirrors of download.elasticsearch.org should be considered, in case of planned or unplanned outages.

I guess maintaining download.elasticsearch.org has limited resources to handle all of these things. I'm afraid the volunteering technical staff will not scale very well with the hopefully ever growing number of Elasticsearch plugins.

@kimchy

This comment has been minimized.

Show comment
Hide comment
@kimchy

kimchy Jan 10, 2013

Member

@jprante agreed, those are good questions you asked for regarding the download service, which we need to figure out if we allow for people to upload plugins to download.elasticsearch.org. For now though, "code plugins" can easily be downloaded from maven as well, so I don't see a problem there (plugin -install <group_id>/<artifact_id>/<version>).

Member

kimchy commented Jan 10, 2013

@jprante agreed, those are good questions you asked for regarding the download service, which we need to figure out if we allow for people to upload plugins to download.elasticsearch.org. For now though, "code plugins" can easily be downloaded from maven as well, so I don't see a problem there (plugin -install <group_id>/<artifact_id>/<version>).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment