Plugins jars are not published to maven central #18131
Comments
|
the phonetic analyzer is just a bit of glue code (exposing a lucene analyzer via ES core), not sure if we need to expose the jar here? I tried to write a plugin and wanted to use painless as a scripting engine for testing, but that one is not exposed either due to being a module. I could have of course mocked my own, but painless looked eas to use, except the dependency missing. |
This seems like a fair reason, though, to be honest, ES core requires that all of its plugins be tested in isolation so maybe we have the same requirements for plugins? I dunno. |
|
First reason is that because we have this JAR generated so we should upload it. Now for my use case, I'm testing an ingest plugin I wrote. I want to test it using a Mock Node which starts:
I can register my plugin in my Mock node but not the geoip one. |
The purpose of deploying to maven is so users can use the software, but plugins do not require a jar in maven.
We are not a library, and especially our plugins are not libraries. Their purpose is to add additional capabilities to elasticsearch, and should remain with that narrow purpose.
Don't do that. We have real integration test capabilities, and if you want to test "with other plugins", then it should be done through those integration tests, which would install the plugin via the zip, not a fantasy land test that drags in a plugin into the classpath. The integ test setup in gradle for adding plugins currently expects a gradle project as input. However, I think we can extend this to take a dependency string, so it would pull the plugin zip from maven. |
Actually! If they expose anything over the java API I think they do? I would love us to get our test tooling working so it can be used from outside the project to test plugins with the zip though. |
That is true, but is only necessary until we have an http client I think? |
I don't know if we can tell people who use a plugin or module that they have to move to the java client earlier than users of just the core so I think we'll need to publish the jars at least until the rest based java client is awesome. I haven't a clue how long that'll take though. |
Do we have plugins that expose additional Java APIs? |
The reindex module and delete-by-query plugin both do. I'm not sure it the ingest-grok module or the other ingest plugins do but they might. I'm fine with only publishing jars for things that add to the java api. |
That's exactly where I was going. I'd argue though that we are making the same mistake we are making in core right now where there is an API that we want to expose, but we produce only a monolithic jar and so end-users end up depending on the Internet that Maven downloads. |
|
IMHO the situation is simple. We are building jar. We assemble then this JAR within a ZIP. If we don't want to publish a JAR, then there is no need to build one. Just copy the classes and the ressources within the ZIP. I don't think we should think instead of devs or users by reducing what they might want to do with what we are building. If a user can start a Node within an integration test he is writing without the need of depending on our test infra, that's perfectly fine. For the case I reported, I'm building a plugin with Maven. I don't want to use the REST tests which are launching an external node. As the plugin developper, it's my decision to run the tests as I want. It can be manual tests with a shell script. It can be tests run from JUnit. Whatever. I'll choose the most efficient tool for me. We used to publish the jars at the same time we were publishing the zip. We are still building the JAR. Is that super complicated to upload the JAR to maven central which is used within the ZIP file? |
Yes and no. It was sort of broken how we were doing it before. We publish the zip and the jar with the same artifact id, and in fact the pom is for the jar. It means when getting the zip, it appears that it has dependencies, when it doesn't really.
This is simply not how plugins work. The way we create child classloaders is by adding jar files. We should not add everything in the zip to the child classloader. |
We could certainly try to separate the client stuff, the common stuff, and the implementation but I don't think it is worth the effort if we expect there will be no more client at some point in the future. I'm ok with doing it how we've been doing it so long as the transport client is a known dead end that we only have to support while we are replacing it. I don't like having to make new things for the transport client but I can live with it, placing great hope in @javanna's work on the rest based java client. I'm not really sure how to respond to @dadoonet's point. Personally I think we should publish the jars so users can do whatever they want with them. I also agree that ESIntegTestCase is a crutch and it'd be great not to have to use it and instead have the build run a separate process. I think that is reasonably easy to do by depending on Elasticsearch's build. But if you for some reason can't or won't use gradle then ESIntegTestCase is really the most convenient thing. And for some things, convenience is more important than doing it right. Actually a lot of the time.... But then again we, Elasticsearch committers, have to keep ESIntegTestCase alive for all that to work. My opinion is still that we should publish the jars and encourage people to use the gradle builds. When we remove the last user of ESIntegTestCase from Elasticsearch we delete it and have a public wake, celebrating its long, glorious life and the great service it did Elasticsearch in 0.90 and 1.x. By then we'll have something better for people to use, or everyone will be using gradle, or it won't matter for some other reason. Or we'll help people. |
|
Discussed in FixItFriday - we should only publish jars that expose an API that the client needs to program against. Once the HTTP client becomes the standard, we won't even need to do that. |
|
Closing as we are publishing the ZIP file on maven central and the outcome of the previous discussion is that we are not going to publish jars but only zip files. BTW I wonder then if it makes sense to publish our plugins on maven central after all. We are publishing them on our download service. May be it's not needed to bring all plugins ZIP to maven central. |
|
As a plugin developer it was extremely convenient to have the jars deployed:
Concerning classloader issue, I totally agree and to be honest I spotted one issue only when I deployed on a test node. But what's the suggested solution here? Do I have to refactor my build system and switch to gradle? Thanks. |
|
As a plugin developer, I am instructing my users to rely on published jars. This is essential for reusability. Not only exposed APIs, but all the classes and resources. Plugin ZIPs are just an additional service for non-developers, but not essential. So I agree with @dadoonet why Elastic is even publishing plugins on Maven Central at all, and not just pointing users to a download zip URL somewhere. I can only see the solution is in copy/paste all the ES plugin code and publish them to Maven Central again under my "org.xbib" group but with all regular Java artifacts. |
|
I started with a "plugin bundle" of mine on Maven Central, see http://search.maven.org/#search%7Cga%7C1%7Cxbib This could be the way to publish ES core plugins too. Sad thing is that current Elasticsearch plugin tool does not find plugin ZIPs from Maven Central generated by a classifier "plugin". But that is not painful to me since I do not use Elasticsearch plugin tool, instead I unzip plugin archives manually and build Docker images. |
|
For the record, we have the same concern with modules. See #16466. |
Plugin zips are published to maven so that they can be tested with. In gradle, the integ test setup allows adding arbitrary extra plugins.
I can add this to the build setup to happen automatically (via a flag in build.gradle) for those modules/plugins that need it. But we will need to use a different artifact id (eg right now in 2.x, we put up a pom for the jar with dependencies, and then the zip just sits alongside it with the same pom, but in 5.x we put up a pom that has no deps and basically is there to mark the zip as being a binary artifact). So I would propose taking the module/plugin name and appending |
|
Sorry for bringing back an old thread. I don't understand why some plugins, maintained in this repository, are published to Maven Central, and some not.
I'm using embedded elasticsearch via spring (spring-boot-starter-data-elasticsearch) and the only way for me to use analysis-phonetic is to use the old stuff, or some 5.0 alpha (unless I would like to manage this dependency manually). |
Embedding elasticsearch is not supported. Read https://www.elastic.co/blog/elasticsearch-the-server If you have more questions, please open a discussion on discuss.elastic.co which the best place for that. |
http://repo1.maven.org/maven2/org/elasticsearch/plugin/analysis-phonetic/5.0.0-alpha2/
We used to publish them before 5.0. See http://repo1.maven.org/maven2/org/elasticsearch/plugin/analysis-phonetic/2.3.2/
Can we add them back?
The text was updated successfully, but these errors were encountered: