Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to gradle build system #13930

Closed
rjernst opened this issue Oct 5, 2015 · 17 comments
Closed

Switch to gradle build system #13930

rjernst opened this issue Oct 5, 2015 · 17 comments

Comments

@rjernst
Copy link
Member

@rjernst rjernst commented Oct 5, 2015

We currently have a very large maven build system, comprising of 6k lines of xml. While maven parent POMs have been very useful in de-duplicating build logic across elasticsearch and plugins, the rigidity of maven has required us to begin implementing most of our test logic through maven-antrun-plugin. Another major problem with multi-module builds is trap: building a module requires explicitly listing an upstream module in order to get a local build of the dependency, or building from a higher level module (which implicitly builds all sub-modules). And finally, the maven model is incapable of decoupling test dependencies from main dependencies, in that tests for module A, cannot depend on module B, which in turn depends on module A (this is not a cycle).

I have a branch which contains elasticsearch switched to using gradle:
https://github.com/elastic/elasticsearch/tree/burn_maven_with_fire

This is a tracking issue for completing the migration to gradle. The tasks are separated into two groups: those remaining required before pushing to master (as to not completely break development), and those required for the migration to be complete, which must happen before the next release (tentatively 2.2).

Required for master push:

  • Maven publishing
  • Windows integ test support
  • Setup publishing of buildSrc (ie ES gradle build plugins)
  • Publish "attach project plugin" (allows dynamically including a multi-project build inside another build)

Required for release:

  • Copyright header check
  • Vagrant qa tests
  • Multi node qa tests (#14921)
  • Rpm/deb building complete
  • Rpm signing
  • Distribution tests
  • Update plugins not in elasticsearch repo to use ES gradle build plugins (mapper attachment: elastic/elasticsearch-mapper-attachments#184)
  • Document new ES gradle build plugins (for ES plugin authors)
  • Document building/testing in README/TESTING (this is done except for jacoco reports)
  • integ tests parity: #14464

Addtional nice to haves:

  • Make qa plugins work in eclipse (need to configure custom classpath)
  • Add run task for elasticsearch and plugins to replace run.sh/run.bat: #14451
@nik9000
Copy link
Contributor

@nik9000 nik9000 commented Oct 5, 2015

Document building/testing in README/TESTING

You should probably add some of this to the pre-merge checklist to. Maybe just the some basic build instructions and a big note saying "![under construction][underconstruction.gif] All references to maven in this file are obsolete and will be replaced with gradle soon. We're very sorry for the inconvenience. ![under construction][underconstruction.gif]"

Document building/testing in README/TESTING
Rpm/deb building complete
Distribution tests
Vagrant qa tests
Rpm signing

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

Document new ES gradle build plugins (for ES plugin authors)

I'm cool with grabbing this later, maybe porting github.com/wikimedia/search-highlighter or something to prove it out.

@rjernst
Copy link
Member Author

@rjernst rjernst commented Oct 5, 2015

@dadoonet
Copy link
Member

@dadoonet dadoonet commented Oct 6, 2015

I was wondering if we can contribute this plugin to Gradle as it sounds like to me a useful one for Gradle community. Is it doable at some point (when stable enough)?

@rjernst
Copy link
Member Author

@rjernst rjernst commented Oct 6, 2015

The gradle folks are working on a longer term solution for combining multiple multi-project builds. This is just a semi-temporary solution.

@dadoonet
Copy link
Member

@dadoonet dadoonet commented Oct 6, 2015

The gradle folks are working on a longer term solution for combining multiple multi-project builds.

Great news!

@nik9000
Copy link
Contributor

@nik9000 nik9000 commented Oct 21, 2015

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

And now that I'm finally done with @Test I can think of helping again! Hurray!

@mrsolo
Copy link
Contributor

@mrsolo mrsolo commented Oct 21, 2015

I would add 'ability to do third party license analysis' as part of release requirement.
currently 3rd party uses license maven plugin
with 'mvn -P license site' call

gradle seems to have one

@rjernst
Copy link
Member Author

@rjernst rjernst commented Oct 21, 2015

@mrsolo afaik, we only use the maven license plugin to check files have a license header?

@mrsolo
Copy link
Contributor

@mrsolo mrsolo commented Oct 21, 2015

@rjernst we use http://code.mycila.com/license-maven-plugin/ to check the license header which is a different license plug in than http://www.mojohaus.org/license-maven-plugin/

we use depends on maven license plugin to do third party library license auditing

@mrsolo
Copy link
Contributor

@mrsolo mrsolo commented Oct 21, 2015

Hmm .. since 2.x, the licenses are managed in https://github.com/elastic/elasticsearch/tree/2.1/distribution/licenses and some if not all libraries are not shaded, this pretty much invalidate pre 2.x auditing workflo, gradle 3rd party dependencies therefore is nice to have but not necessary.

Auditing will be

  1. Cross check content of distribution *.zip against information under distribution/licenses
  2. Generate information for external consumption
@rjernst
Copy link
Member Author

@rjernst rjernst commented Oct 21, 2015

#1 is already done in the port of the license checker (perl script on master). Not sure what you mean by #2

@mrsolo
Copy link
Contributor

@mrsolo mrsolo commented Oct 21, 2015

#2 basically mean generation of google spreadsheet, currently not automated

rjernst added a commit that referenced this issue Oct 29, 2015
rjernst added a commit to rjernst/elasticsearch that referenced this issue Oct 30, 2015
The test jar was previously built in maven by copying class files. With
gradle we now have a proper test framework artifact. This change moves
the classes used by the test framework into the test-framework module.

See elastic#13930
rjernst added a commit to rjernst/elasticsearch that referenced this issue Oct 30, 2015
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See elastic#13930
rjernst added a commit to rjernst/elasticsearch that referenced this issue Oct 30, 2015
The randomized testing gradle code belongs with the randomized testing
library. This moves it to its own repository, which is now published as
a snapshot in nexus.

See elastic#13930
@nik9000
Copy link
Contributor

@nik9000 nik9000 commented Oct 30, 2015

Ok - I'm going to look at the vagrant tests.

@nik9000
Copy link
Contributor

@nik9000 nik9000 commented Nov 5, 2015

Ok - I'm going to look at the vagrant tests.

I've got the vagrant tests working as well as I can without rpm and deb packages so I'm going to start work on fixing the deb package.

@nik9000
Copy link
Contributor

@nik9000 nik9000 commented Nov 18, 2015

except for instructions for vagrant tests and jacoco reports

Ok - I'm going to see about these.

@spinscale spinscale added v2.3.0 and removed v2.2.0 labels Dec 23, 2015
karmi added a commit to elastic/elasticsearch-ruby that referenced this issue Jan 9, 2016
Elasticsearch switched to Gradle as the build system in
elastic/elasticsearch#13930,
and the current Maven-oriented script stopped working.

This patch refactors the code and adds a better infrastructure
for future expansion.
@clintongormley
Copy link
Contributor

@clintongormley clintongormley commented Jan 28, 2016

This is done, bar the docs for plugin authors (#15280). Closing

@agentgt
Copy link

@agentgt agentgt commented Mar 13, 2019

@rjernst I'm curious did you not know about <relativePath> in Maven. I looked at the previous versions of the project you don't include <relativePath> which makes me think your entire hate for Maven could have stemmed from not knowing this feature.

From maven: https://maven.apache.org/pom.html#Inheritance

"Notice the relativePath element. It is not required, but may be used as a signifier to Maven to first search the path given for this project's parent, before searching the local and then remote repositories."

Thus if you had put the relative path to your parent pom in your submodules maven would not have to go "download" the internet.

I only bring this up because I have people point to your gradle video about how the maven reactor won't let them build a part of the project without downloading other modules and thus fundamentally flawed.

EDIT: Oh I thought your parent was in a non standard directory. I guess you mean it goes and downloads snapshot pom files. There are ways around that but yes that is the reactor behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.