Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[maven] split packaging into modules #11523

Closed
wants to merge 21 commits into from

Conversation

@dadoonet
Copy link
Member

dadoonet commented Jun 5, 2015

Important: don't merge this PR as is. It requires more testing.
I'm opening it only to let people discuss on it and share thoughts.

The idea is to have elasticsearch core project responsible of providing a JAR and have other modules packaging this JAR and some dependencies to produce zip, tar.gz, deb and rpm.

This change creates a proper distribution modules in which we have today packaging for:

  • zip
  • tar.gz
  • rpm
  • deb

It creates a new /distribution dir in elasticsearch project.
This one contains:

├── deb
│   └── src
│       └── packaging
│           ├── init.d
│           ├── lintian
│           └── scripts
├── rpm
│   └── src
│       └── packaging
│           └── init.d
├── src
│   ├── main
│   │   ├── assemblies
│   │   └── resources
│   │       ├── bin
│   │       ├── config
│   │       └── lib
│   └── packaging
│       ├── env
│       ├── scripts
│       └── systemd
│           └── sysctl
├── tar
│   └── src
│       └── main
│           └── assemblies
└── zip
    └── src
        └── main
            └── assemblies

And then, if you run cd distribution; mvn clean install, you get:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Elasticsearch Distribution ........................ SUCCESS [5.665s]
[INFO] Elasticsearch ZIP Distribution .................... SUCCESS [3.389s]
[INFO] Elasticsearch TAR Distribution .................... SUCCESS [2.621s]
[INFO] Elasticsearch DEB Distribution .................... SUCCESS [2.481s]
[INFO] Elasticsearch RPM Distribution .................... SUCCESS [0.842s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Adding a new packaging should be adding a new module and define in its pom.xml any specific needs.

@tlrx
Copy link
Member

tlrx commented Jun 11, 2015

@dadoonet nice work! Here is a first round of comments:

  1. I don't think that the package names should be changed (before: elasticsearch-2.0.0-SNAPSHOT.zip, in this pull request: elasticsearch-zip-2.0.0-SNAPSHOT.zip). Same thing for Deb/Tar/Zip packages.
  2. Classpath in elasticsearch.in.* scripts is modified and does not seem right (before: ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-2.0.0-SNAPSHOT.jar:..., in this pull request: ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-tar-2.0.0-SNAPSHOT.jar:...
  3. There's something I don't get about libs. The current master core project generates Zip/TarGz packages that includes many libs (ex: netty, commons-cli, jackson...). But the 1.6 Zip/TarGz does not include those libs, neither the Deb package generated by this pull request...
  4. /elasticsearch/core/src/packaging directory does not seem to be removed... Or maybe it's just my IDE?
  5. That would be great if /elasticsearch/distribution/src could be renamed into /elasticsearch/distribution/common because it really contains common files between packaging types.
  6. I can't generate the RPM package, command mvn clean package rpm:rpm -DskipTests -Dgpg.passphrase=none produces error message Source location /home/tanguy/Github/elasticsearch/distribution/rpm/config does not exist
  7. I supposed the release script will need to be updated as well since artefacts will be located in different places.

Once those points are resolved, I'll be able to run automated tests for each packaging types.

@tlrx
Copy link
Member

tlrx commented Jun 15, 2015

@dadoonet Point 3 is resolved by #11664

@dadoonet dadoonet force-pushed the dadoonet:maven/distribution branch Jun 15, 2015
@dadoonet
Copy link
Member Author

dadoonet commented Jun 15, 2015

@tlrx I updated with some new commits.

I fixed points 1, 2, 4

5. I used maven convention here so every source file should be under src/.
6. I tried to run the same but I don't have Ubuntu running. I guess I should try to launch a VM on my machine and run that from it?
7. Yes. That's is still on the TODO list (same for plugins BTW).

Let me know what you think about the new commits.

@dadoonet dadoonet force-pushed the dadoonet:maven/distribution branch 2 times, most recently Jun 16, 2015
@dadoonet
Copy link
Member Author

dadoonet commented Jun 22, 2015

@tlrx any chance you could review this again? It's a blocker for #11587. Thanks!

@dadoonet dadoonet force-pushed the dadoonet:maven/distribution branch Jun 29, 2015
@tlrx
Copy link
Member

tlrx commented Jun 30, 2015

@dadoonet great work, I think we're getting close to terminate this.

I tested TAR and DEB packages on Debian 7.8, Debian 8, Ubuntu 12.04, Ubuntu 14.04 and Ubuntu 15.04: it works like a charm.

ZIP package is identical to TAR so I skipped it.

RPM can now be built with the mvn install command but it has a wrong naming: elasticsearch-rpm-2.0.0-SNAPSHOT20150630120151.noarch.rpm instead of elasticsearch-2.0.0-....rpm` which makes it impossible for me to run the automated tests. I checked its content and everything looks fine except the file name.

I also noticed that the jsr166e-1.1.0.jar is missing in the packages but exists right now in master... maybe it has been added in the meanwhile?

@dadoonet
Copy link
Member Author

dadoonet commented Jun 30, 2015

RPM can now be built with the mvn install command but it has a wrong naming: elasticsearch-rpm-2.0.0-SNAPSHOT20150630120151.noarch.rpm instead of elasticsearch-2.0.0-....rpm` which makes it impossible for me to run the automated tests. I checked its content and everything looks fine except the file name.

Argh! Good catch. Going to fix.

I also noticed that the jsr166e-1.1.0.jar is missing in the packages but exists right now in master... maybe it has been added in the meanwhile?

Ah! yeah. I probably missed something! Thank you so much for catching it.

Also, I'll need to rebase again my work because new changes have been added to RPM in master. It will require some other tests.

BTW: I'm looking for volunteers to also update the release script when this PR will be merged in or by forking my branch and contributing there. We will need after this PR to look for RPM/DEB/ZIP/TARGZ in their own distribution/xxx/target/release dirs and also adapt the signed/unsigned RPM stuff...

I might be able to look at it but probably not before thursday morning.

@dadoonet
Copy link
Member Author

dadoonet commented Jul 1, 2015

@tlrx I did some change and some new tests based on your feedback.
Sounds like we are making big progress! :)

This new change now publish an artifact named elasticsearch-2.0.0-SNAPSHOT.rpm but this name is only set when maven copy the artifact to .m2. Not sure yet how I can copy it back to target/releases as this sounds our convention.

That being said, I'm curious to see if the one in the .m2 can pass your tests.

BTW, I also added an icon for the RPM :)

@dadoonet
Copy link
Member Author

dadoonet commented Jul 3, 2015

@tlrx ping! I'd really like to move that forward and merge it soonish. So if you can test it or share with me your test scripts that would be awesome!

@dadoonet
Copy link
Member Author

dadoonet commented Jul 3, 2015

Just adding for reference here a pending change that needs to be applied here as well if merged: #12010

@tlrx
Copy link
Member

tlrx commented Jul 3, 2015

@dadoonet On my laptop the RPM package still has a wrong filename:

  • Maven project:
    • elasticsearch-2.0.0-SNAPSHOT.rpm-2.0.0-SNAPSHOT20150703093546.noarch.rpm
  • Maven local repository:
    • elasticsearch-rpm-2.0.0-SNAPSHOT.rpm

If we change <name>elasticsearch-${project.version}.rpm</name> to <name>elasticsearch</name> in the pom.xml, we end with:

  • Maven project:
    • elasticsearch-2.0.0-SNAPSHOT20150703094032.noarch.rpm << ok
  • Maven local repository:
    • elasticsearch-rpm-2.0.0-SNAPSHOT.rpm << wrong

It seems difficult to have a coherent naming between the built artefact and the installed artefact.

With this last change in the pom.xml, RPM works on Fedora21, CentOS6.6, CentOS7,OpenSUSE13 (I did not test tar.gz/zip/deb packages)

The jsr166e-1.1.0.jar is still missing in the packages... did you find out if it has been added in master?

@dadoonet dadoonet force-pushed the dadoonet:maven/distribution branch Jul 6, 2015
@dadoonet
Copy link
Member Author

dadoonet commented Jul 6, 2015

@tlrx Could you try to build my branch again? I don't understand why my VM does not accept it anymore and fails...

@spinscale
Copy link
Member

spinscale commented Jul 6, 2015

any trick to build the RPM as part of mvn package in the root dir? Is there anything specific that fails for you or should be tested?

@spinscale
Copy link
Member

spinscale commented Jul 6, 2015

@clintongormley @s1monw I think that this one is a requirement to split out the plugin manager into its own sub-project. Reason for this is, that the plugin-manager source has a dependency to the core, but when packaging everything together, the plugin manager jar needs to be included in the distribution. I do not see a good way doing this, when creating the different packages is part of the core building process.

@dadoonet
Copy link
Member Author

dadoonet commented Jul 6, 2015

@spinscale if your have rpm installed on your machine, then when running mvn package, rpm module is automatically added to the build.

See https://github.com/dadoonet/elasticsearch/blob/maven/distribution/distribution/pom.xml#L152-163

I tested /usr/bin/rpmbuild. Do you have it on your machine?

@tlrx
Copy link
Member

tlrx commented Jul 6, 2015

@tlrx Could you try to build my branch again? I don't understand why my VM does not accept it anymore and fails...

I can build it on my laptop (commit dcc34b5e672204e86b5452ded1db8ba34e1a3d1d).

@spinscale
Copy link
Member

spinscale commented Jul 6, 2015

@dadoonet installed rpm via brew on osx, thus it is /usr/local/bin/rpmbuild

@dadoonet
Copy link
Member Author

dadoonet commented Jul 6, 2015

@spinscale oh? I was not aware that we can have rpm on OSX! I spent too much time trying to work with a VM! I guess you saved me here! Thanks. So I'll will adapt the PR to check in both dirs. For now, you can edit the pom.xml file and link to your dir.

@spinscale
Copy link
Member

spinscale commented Jul 6, 2015

maybe have a property as well ${packaging.rpm.rpmbuild}, so one can configure via the commandline and no need to change files in case it is somewhere else?

dadoonet added 12 commits Jun 29, 2015
... when `/usr/bin/rpmbuild` is available on the machine
It was previously generating a fake elasticsearch-x.x.x.jar and was adding it twice in the debian artifact.
We can use `rpm` as a packaging type in maven.
It allows to automatically attach the generated rpm to the build cycle and add it as a generated artifact. So it's also published in the `.m2` and in maven central as well.
Also disable rpm module when not on a RPM compatible machine
@dadoonet dadoonet force-pushed the dadoonet:maven/distribution branch to faa64c6 Jul 7, 2015
@dadoonet
Copy link
Member Author

dadoonet commented Jul 7, 2015

@spinscale I rebased, fixed the deb naming issue and also move rpm plugin to 2.1.3 as you did in a previous change.
Will work on fixing rpm working as well on MacOSX now and other remaining stuff.

dadoonet added 2 commits Jul 7, 2015
And also support definition of any other path for rpmbuild
@dadoonet
Copy link
Member Author

dadoonet commented Jul 7, 2015

I added 2 new changes:

  • RPM can now work also on MacOSX (thanks @spinscale for telling me I can run brew install rpm)
  • rpm final file is also copied with the right name under target/releases.

Note that when artifact SHA1 will be computed, we need to make sure this is still working as expected.

That being said, may be we should simply adapt our release tool to publish everything which has been installed in the local ~/.m2 as those are the right files... I think (but unsure) that mvn deploy actually deploy files from this ~/.m2 dir and not from target or target/releases.

@tlrx @spinscale Could you look at this please?

Also if you can share with me what exact commands you are using to test deb packages that will be easier for me to test and fix instead of pinging you all the time :).

@spinscale
Copy link
Member

spinscale commented Jul 13, 2015

ok, two things. I needed to fix the packages to contain the working directory for start-stop-daemon and systemd. This is the diff

diff --git a/distribution/deb/src/main/packaging/init.d/elasticsearch b/distribution/deb/src/main/packaging/init.d/elasticsearch
index 0c5e91d..19198c9 100755
--- a/distribution/deb/src/main/packaging/init.d/elasticsearch
+++ b/distribution/deb/src/main/packaging/init.d/elasticsearch
@@ -173,7 +173,7 @@ case "$1" in
        fi

        # Start Daemon
-       start-stop-daemon --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
+       start-stop-daemon -d $ES_HOME --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
        return=$?
        if [ $return -eq 0 ]
        then
diff --git a/distribution/src/main/packaging/systemd/elasticsearch.service b/distribution/src/main/packaging/systemd/elasticsearch.service
index a4c2699..b857d61 100644
--- a/distribution/src/main/packaging/systemd/elasticsearch.service
+++ b/distribution/src/main/packaging/systemd/elasticsearch.service
@@ -13,6 +13,8 @@ Environment=LOG_DIR=${packaging.elasticsearch.log.dir}
 Environment=PID_DIR=${packaging.elasticsearch.pid.dir}
 EnvironmentFile=-${packaging.env.file}

+WorkingDirectory=${packaging.elasticsearch.home.dir}
+
 User=${packaging.elasticsearch.user}
 Group=${packaging.elasticsearch.group}

One bigger issue remains. The distribution/pom.xml and all the pom files below contains a SNAPSHOT parent and each sub-project contains a version of the distribution pom. Who is updating those to stable values and ensure that they do not run out of sync?

    <parent>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-parent</artifactId>
        <version>2.0.0-SNAPSHOT</version>
    </parent>

// each sub project pom.xml
    <parent>
        <groupId>org.elasticsearch.distribution</groupId>
        <artifactId>elasticsearch-distribution</artifactId>
        <version>2.0.0-SNAPSHOT</version>
    </parent>

Apart from that I tested the packages and they look good.

@spinscale
Copy link
Member

spinscale commented Jul 27, 2015

obsoleted by #12286

@spinscale spinscale closed this Jul 27, 2015
@dadoonet dadoonet deleted the dadoonet:maven/distribution branch Aug 18, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.