Switch to gradle build system #13930

Closed
rjernst opened this Issue Oct 5, 2015 · 16 comments

Comments

Projects
None yet
6 participants
@rjernst
Member

rjernst commented Oct 5, 2015

We currently have a very large maven build system, comprising of 6k lines of xml. While maven parent POMs have been very useful in de-duplicating build logic across elasticsearch and plugins, the rigidity of maven has required us to begin implementing most of our test logic through maven-antrun-plugin. Another major problem with multi-module builds is trap: building a module requires explicitly listing an upstream module in order to get a local build of the dependency, or building from a higher level module (which implicitly builds all sub-modules). And finally, the maven model is incapable of decoupling test dependencies from main dependencies, in that tests for module A, cannot depend on module B, which in turn depends on module A (this is not a cycle).

I have a branch which contains elasticsearch switched to using gradle:
https://github.com/elastic/elasticsearch/tree/burn_maven_with_fire

This is a tracking issue for completing the migration to gradle. The tasks are separated into two groups: those remaining required before pushing to master (as to not completely break development), and those required for the migration to be complete, which must happen before the next release (tentatively 2.2).

Required for master push:

  • Maven publishing
  • Windows integ test support
  • Setup publishing of buildSrc (ie ES gradle build plugins)
  • Publish "attach project plugin" (allows dynamically including a multi-project build inside another build)

Required for release:

  • Copyright header check
  • Vagrant qa tests
  • Multi node qa tests (#14921)
  • Rpm/deb building complete
  • Rpm signing
  • Distribution tests
  • Update plugins not in elasticsearch repo to use ES gradle build plugins (mapper attachment: elastic/elasticsearch-mapper-attachments#184)
  • Document new ES gradle build plugins (for ES plugin authors)
  • Document building/testing in README/TESTING (this is done except for jacoco reports)
  • integ tests parity: #14464

Addtional nice to haves:

  • Make qa plugins work in eclipse (need to configure custom classpath)
  • Add run task for elasticsearch and plugins to replace run.sh/run.bat: #14451
@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Oct 5, 2015

Contributor

Document building/testing in README/TESTING

You should probably add some of this to the pre-merge checklist to. Maybe just the some basic build instructions and a big note saying "![under construction][underconstruction.gif] All references to maven in this file are obsolete and will be replaced with gradle soon. We're very sorry for the inconvenience. ![under construction][underconstruction.gif]"

Document building/testing in README/TESTING
Rpm/deb building complete
Distribution tests
Vagrant qa tests
Rpm signing

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

Document new ES gradle build plugins (for ES plugin authors)

I'm cool with grabbing this later, maybe porting github.com/wikimedia/search-highlighter or something to prove it out.

Contributor

nik9000 commented Oct 5, 2015

Document building/testing in README/TESTING

You should probably add some of this to the pre-merge checklist to. Maybe just the some basic build instructions and a big note saying "![under construction][underconstruction.gif] All references to maven in this file are obsolete and will be replaced with gradle soon. We're very sorry for the inconvenience. ![under construction][underconstruction.gif]"

Document building/testing in README/TESTING
Rpm/deb building complete
Distribution tests
Vagrant qa tests
Rpm signing

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

Document new ES gradle build plugins (for ES plugin authors)

I'm cool with grabbing this later, maybe porting github.com/wikimedia/search-highlighter or something to prove it out.

@rjernst

This comment has been minimized.

Show comment
Hide comment
@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Oct 6, 2015

Member

I was wondering if we can contribute this plugin to Gradle as it sounds like to me a useful one for Gradle community. Is it doable at some point (when stable enough)?

Member

dadoonet commented Oct 6, 2015

I was wondering if we can contribute this plugin to Gradle as it sounds like to me a useful one for Gradle community. Is it doable at some point (when stable enough)?

@rjernst

This comment has been minimized.

Show comment
Hide comment
@rjernst

rjernst Oct 6, 2015

Member

The gradle folks are working on a longer term solution for combining multiple multi-project builds. This is just a semi-temporary solution.

Member

rjernst commented Oct 6, 2015

The gradle folks are working on a longer term solution for combining multiple multi-project builds. This is just a semi-temporary solution.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Oct 6, 2015

Member

The gradle folks are working on a longer term solution for combining multiple multi-project builds.

Great news!

Member

dadoonet commented Oct 6, 2015

The gradle folks are working on a longer term solution for combining multiple multi-project builds.

Great news!

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Oct 21, 2015

Contributor

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

And now that I'm finally done with @Test I can think of helping again! Hurray!

Contributor

nik9000 commented Oct 21, 2015

I'm cool with grabbing these later this week. If anyone that wants them before I get to them then please take them with my best wishes.

And now that I'm finally done with @Test I can think of helping again! Hurray!

@mrsolo

This comment has been minimized.

Show comment
Hide comment
@mrsolo

mrsolo Oct 21, 2015

Contributor

I would add 'ability to do third party license analysis' as part of release requirement.
currently 3rd party uses license maven plugin
with 'mvn -P license site' call

gradle seems to have one

Contributor

mrsolo commented Oct 21, 2015

I would add 'ability to do third party license analysis' as part of release requirement.
currently 3rd party uses license maven plugin
with 'mvn -P license site' call

gradle seems to have one

@rjernst

This comment has been minimized.

Show comment
Hide comment
@rjernst

rjernst Oct 21, 2015

Member

@mrsolo afaik, we only use the maven license plugin to check files have a license header?

Member

rjernst commented Oct 21, 2015

@mrsolo afaik, we only use the maven license plugin to check files have a license header?

@mrsolo

This comment has been minimized.

Show comment
Hide comment
@mrsolo

mrsolo Oct 21, 2015

Contributor

@rjernst we use http://code.mycila.com/license-maven-plugin/ to check the license header which is a different license plug in than http://www.mojohaus.org/license-maven-plugin/

we use depends on maven license plugin to do third party library license auditing

Contributor

mrsolo commented Oct 21, 2015

@rjernst we use http://code.mycila.com/license-maven-plugin/ to check the license header which is a different license plug in than http://www.mojohaus.org/license-maven-plugin/

we use depends on maven license plugin to do third party library license auditing

@mrsolo

This comment has been minimized.

Show comment
Hide comment
@mrsolo

mrsolo Oct 21, 2015

Contributor

Hmm .. since 2.x, the licenses are managed in https://github.com/elastic/elasticsearch/tree/2.1/distribution/licenses and some if not all libraries are not shaded, this pretty much invalidate pre 2.x auditing workflo, gradle 3rd party dependencies therefore is nice to have but not necessary.

Auditing will be

  1. Cross check content of distribution *.zip against information under distribution/licenses
  2. Generate information for external consumption
Contributor

mrsolo commented Oct 21, 2015

Hmm .. since 2.x, the licenses are managed in https://github.com/elastic/elasticsearch/tree/2.1/distribution/licenses and some if not all libraries are not shaded, this pretty much invalidate pre 2.x auditing workflo, gradle 3rd party dependencies therefore is nice to have but not necessary.

Auditing will be

  1. Cross check content of distribution *.zip against information under distribution/licenses
  2. Generate information for external consumption
@rjernst

This comment has been minimized.

Show comment
Hide comment
@rjernst

rjernst Oct 21, 2015

Member

#1 is already done in the port of the license checker (perl script on master). Not sure what you mean by #2

Member

rjernst commented Oct 21, 2015

#1 is already done in the port of the license checker (perl script on master). Not sure what you mean by #2

@mrsolo

This comment has been minimized.

Show comment
Hide comment
@mrsolo

mrsolo Oct 21, 2015

Contributor

#2 basically mean generation of google spreadsheet, currently not automated

Contributor

mrsolo commented Oct 21, 2015

#2 basically mean generation of google spreadsheet, currently not automated

rjernst added a commit that referenced this issue Oct 29, 2015

rjernst added a commit to rjernst/elasticsearch that referenced this issue Oct 30, 2015

Build: Move test framework files to their new location
The test jar was previously built in maven by copying class files. With
gradle we now have a proper test framework artifact. This change moves
the classes used by the test framework into the test-framework module.

See #13930

rjernst added a commit to rjernst/elasticsearch that referenced this issue Oct 30, 2015

Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930

rjernst added a commit to rjernst/elasticsearch that referenced this issue Oct 30, 2015

Build: Move randomized runner gradle plugin to external dependency
The randomized testing gradle code belongs with the randomized testing
library. This moves it to its own repository, which is now published as
a snapshot in nexus.

See #13930
@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Oct 30, 2015

Contributor

Ok - I'm going to look at the vagrant tests.

Contributor

nik9000 commented Oct 30, 2015

Ok - I'm going to look at the vagrant tests.

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Nov 5, 2015

Contributor

Ok - I'm going to look at the vagrant tests.

I've got the vagrant tests working as well as I can without rpm and deb packages so I'm going to start work on fixing the deb package.

Contributor

nik9000 commented Nov 5, 2015

Ok - I'm going to look at the vagrant tests.

I've got the vagrant tests working as well as I can without rpm and deb packages so I'm going to start work on fixing the deb package.

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Nov 18, 2015

Contributor

except for instructions for vagrant tests and jacoco reports

Ok - I'm going to see about these.

Contributor

nik9000 commented Nov 18, 2015

except for instructions for vagrant tests and jacoco reports

Ok - I'm going to see about these.

@spinscale spinscale added v2.3.0 and removed v2.2.0 labels Dec 23, 2015

karmi added a commit to elastic/elasticsearch-ruby that referenced this issue Jan 9, 2016

Updated the `elasticsearch:build` Rake task to work with current Gradle
Elasticsearch switched to Gradle as the build system in
elastic/elasticsearch#13930,
and the current Maven-oriented script stopped working.

This patch refactors the code and adds a better infrastructure
for future expansion.
@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Jan 28, 2016

Member

This is done, bar the docs for plugin authors (#15280). Closing

Member

clintongormley commented Jan 28, 2016

This is done, bar the docs for plugin authors (#15280). Closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment