[maven] split packaging into modules #11523

Closed
wants to merge 21 commits into
from

Conversation

Projects
None yet
3 participants
@dadoonet
Member

dadoonet commented Jun 5, 2015

Important: don't merge this PR as is. It requires more testing.
I'm opening it only to let people discuss on it and share thoughts.

The idea is to have elasticsearch core project responsible of providing a JAR and have other modules packaging this JAR and some dependencies to produce zip, tar.gz, deb and rpm.

This change creates a proper distribution modules in which we have today packaging for:

  • zip
  • tar.gz
  • rpm
  • deb

It creates a new /distribution dir in elasticsearch project.
This one contains:

├── deb
│   └── src
│       └── packaging
│           ├── init.d
│           ├── lintian
│           └── scripts
├── rpm
│   └── src
│       └── packaging
│           └── init.d
├── src
│   ├── main
│   │   ├── assemblies
│   │   └── resources
│   │       ├── bin
│   │       ├── config
│   │       └── lib
│   └── packaging
│       ├── env
│       ├── scripts
│       └── systemd
│           └── sysctl
├── tar
│   └── src
│       └── main
│           └── assemblies
└── zip
    └── src
        └── main
            └── assemblies

And then, if you run cd distribution; mvn clean install, you get:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Elasticsearch Distribution ........................ SUCCESS [5.665s]
[INFO] Elasticsearch ZIP Distribution .................... SUCCESS [3.389s]
[INFO] Elasticsearch TAR Distribution .................... SUCCESS [2.621s]
[INFO] Elasticsearch DEB Distribution .................... SUCCESS [2.481s]
[INFO] Elasticsearch RPM Distribution .................... SUCCESS [0.842s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Adding a new packaging should be adding a new module and define in its pom.xml any specific needs.

@tlrx

This comment has been minimized.

Show comment
Hide comment
@tlrx

tlrx Jun 11, 2015

Member

@dadoonet nice work! Here is a first round of comments:

  1. I don't think that the package names should be changed (before: elasticsearch-2.0.0-SNAPSHOT.zip, in this pull request: elasticsearch-zip-2.0.0-SNAPSHOT.zip). Same thing for Deb/Tar/Zip packages.
  2. Classpath in elasticsearch.in.* scripts is modified and does not seem right (before: ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-2.0.0-SNAPSHOT.jar:..., in this pull request: ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-tar-2.0.0-SNAPSHOT.jar:...
  3. There's something I don't get about libs. The current master core project generates Zip/TarGz packages that includes many libs (ex: netty, commons-cli, jackson...). But the 1.6 Zip/TarGz does not include those libs, neither the Deb package generated by this pull request...
  4. /elasticsearch/core/src/packaging directory does not seem to be removed... Or maybe it's just my IDE?
  5. That would be great if /elasticsearch/distribution/src could be renamed into /elasticsearch/distribution/common because it really contains common files between packaging types.
  6. I can't generate the RPM package, command mvn clean package rpm:rpm -DskipTests -Dgpg.passphrase=none produces error message Source location /home/tanguy/Github/elasticsearch/distribution/rpm/config does not exist
  7. I supposed the release script will need to be updated as well since artefacts will be located in different places.

Once those points are resolved, I'll be able to run automated tests for each packaging types.

Member

tlrx commented Jun 11, 2015

@dadoonet nice work! Here is a first round of comments:

  1. I don't think that the package names should be changed (before: elasticsearch-2.0.0-SNAPSHOT.zip, in this pull request: elasticsearch-zip-2.0.0-SNAPSHOT.zip). Same thing for Deb/Tar/Zip packages.
  2. Classpath in elasticsearch.in.* scripts is modified and does not seem right (before: ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-2.0.0-SNAPSHOT.jar:..., in this pull request: ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-tar-2.0.0-SNAPSHOT.jar:...
  3. There's something I don't get about libs. The current master core project generates Zip/TarGz packages that includes many libs (ex: netty, commons-cli, jackson...). But the 1.6 Zip/TarGz does not include those libs, neither the Deb package generated by this pull request...
  4. /elasticsearch/core/src/packaging directory does not seem to be removed... Or maybe it's just my IDE?
  5. That would be great if /elasticsearch/distribution/src could be renamed into /elasticsearch/distribution/common because it really contains common files between packaging types.
  6. I can't generate the RPM package, command mvn clean package rpm:rpm -DskipTests -Dgpg.passphrase=none produces error message Source location /home/tanguy/Github/elasticsearch/distribution/rpm/config does not exist
  7. I supposed the release script will need to be updated as well since artefacts will be located in different places.

Once those points are resolved, I'll be able to run automated tests for each packaging types.

@tlrx

This comment has been minimized.

Show comment
Hide comment
@tlrx

tlrx Jun 15, 2015

Member

@dadoonet Point 3 is resolved by #11664

Member

tlrx commented Jun 15, 2015

@dadoonet Point 3 is resolved by #11664

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jun 15, 2015

Member

@tlrx I updated with some new commits.

I fixed points 1, 2, 4

5. I used maven convention here so every source file should be under src/.
6. I tried to run the same but I don't have Ubuntu running. I guess I should try to launch a VM on my machine and run that from it?
7. Yes. That's is still on the TODO list (same for plugins BTW).

Let me know what you think about the new commits.

Member

dadoonet commented Jun 15, 2015

@tlrx I updated with some new commits.

I fixed points 1, 2, 4

5. I used maven convention here so every source file should be under src/.
6. I tried to run the same but I don't have Ubuntu running. I guess I should try to launch a VM on my machine and run that from it?
7. Yes. That's is still on the TODO list (same for plugins BTW).

Let me know what you think about the new commits.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jun 22, 2015

Member

@tlrx any chance you could review this again? It's a blocker for #11587. Thanks!

Member

dadoonet commented Jun 22, 2015

@tlrx any chance you could review this again? It's a blocker for #11587. Thanks!

@tlrx

This comment has been minimized.

Show comment
Hide comment
@tlrx

tlrx Jun 30, 2015

Member

@dadoonet great work, I think we're getting close to terminate this.

I tested TAR and DEB packages on Debian 7.8, Debian 8, Ubuntu 12.04, Ubuntu 14.04 and Ubuntu 15.04: it works like a charm.

ZIP package is identical to TAR so I skipped it.

RPM can now be built with the mvn install command but it has a wrong naming: elasticsearch-rpm-2.0.0-SNAPSHOT20150630120151.noarch.rpm instead of elasticsearch-2.0.0-....rpm` which makes it impossible for me to run the automated tests. I checked its content and everything looks fine except the file name.

I also noticed that the jsr166e-1.1.0.jar is missing in the packages but exists right now in master... maybe it has been added in the meanwhile?

Member

tlrx commented Jun 30, 2015

@dadoonet great work, I think we're getting close to terminate this.

I tested TAR and DEB packages on Debian 7.8, Debian 8, Ubuntu 12.04, Ubuntu 14.04 and Ubuntu 15.04: it works like a charm.

ZIP package is identical to TAR so I skipped it.

RPM can now be built with the mvn install command but it has a wrong naming: elasticsearch-rpm-2.0.0-SNAPSHOT20150630120151.noarch.rpm instead of elasticsearch-2.0.0-....rpm` which makes it impossible for me to run the automated tests. I checked its content and everything looks fine except the file name.

I also noticed that the jsr166e-1.1.0.jar is missing in the packages but exists right now in master... maybe it has been added in the meanwhile?

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jun 30, 2015

Member

RPM can now be built with the mvn install command but it has a wrong naming: elasticsearch-rpm-2.0.0-SNAPSHOT20150630120151.noarch.rpm instead of elasticsearch-2.0.0-....rpm` which makes it impossible for me to run the automated tests. I checked its content and everything looks fine except the file name.

Argh! Good catch. Going to fix.

I also noticed that the jsr166e-1.1.0.jar is missing in the packages but exists right now in master... maybe it has been added in the meanwhile?

Ah! yeah. I probably missed something! Thank you so much for catching it.

Also, I'll need to rebase again my work because new changes have been added to RPM in master. It will require some other tests.

BTW: I'm looking for volunteers to also update the release script when this PR will be merged in or by forking my branch and contributing there. We will need after this PR to look for RPM/DEB/ZIP/TARGZ in their own distribution/xxx/target/release dirs and also adapt the signed/unsigned RPM stuff...

I might be able to look at it but probably not before thursday morning.

Member

dadoonet commented Jun 30, 2015

RPM can now be built with the mvn install command but it has a wrong naming: elasticsearch-rpm-2.0.0-SNAPSHOT20150630120151.noarch.rpm instead of elasticsearch-2.0.0-....rpm` which makes it impossible for me to run the automated tests. I checked its content and everything looks fine except the file name.

Argh! Good catch. Going to fix.

I also noticed that the jsr166e-1.1.0.jar is missing in the packages but exists right now in master... maybe it has been added in the meanwhile?

Ah! yeah. I probably missed something! Thank you so much for catching it.

Also, I'll need to rebase again my work because new changes have been added to RPM in master. It will require some other tests.

BTW: I'm looking for volunteers to also update the release script when this PR will be merged in or by forking my branch and contributing there. We will need after this PR to look for RPM/DEB/ZIP/TARGZ in their own distribution/xxx/target/release dirs and also adapt the signed/unsigned RPM stuff...

I might be able to look at it but probably not before thursday morning.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 1, 2015

Member

@tlrx I did some change and some new tests based on your feedback.
Sounds like we are making big progress! :)

This new change now publish an artifact named elasticsearch-2.0.0-SNAPSHOT.rpm but this name is only set when maven copy the artifact to .m2. Not sure yet how I can copy it back to target/releases as this sounds our convention.

That being said, I'm curious to see if the one in the .m2 can pass your tests.

BTW, I also added an icon for the RPM :)

Member

dadoonet commented Jul 1, 2015

@tlrx I did some change and some new tests based on your feedback.
Sounds like we are making big progress! :)

This new change now publish an artifact named elasticsearch-2.0.0-SNAPSHOT.rpm but this name is only set when maven copy the artifact to .m2. Not sure yet how I can copy it back to target/releases as this sounds our convention.

That being said, I'm curious to see if the one in the .m2 can pass your tests.

BTW, I also added an icon for the RPM :)

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 3, 2015

Member

@tlrx ping! I'd really like to move that forward and merge it soonish. So if you can test it or share with me your test scripts that would be awesome!

Member

dadoonet commented Jul 3, 2015

@tlrx ping! I'd really like to move that forward and merge it soonish. So if you can test it or share with me your test scripts that would be awesome!

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 3, 2015

Member

Just adding for reference here a pending change that needs to be applied here as well if merged: #12010

Member

dadoonet commented Jul 3, 2015

Just adding for reference here a pending change that needs to be applied here as well if merged: #12010

@tlrx

This comment has been minimized.

Show comment
Hide comment
@tlrx

tlrx Jul 3, 2015

Member

@dadoonet On my laptop the RPM package still has a wrong filename:

  • Maven project:
    • elasticsearch-2.0.0-SNAPSHOT.rpm-2.0.0-SNAPSHOT20150703093546.noarch.rpm
  • Maven local repository:
    • elasticsearch-rpm-2.0.0-SNAPSHOT.rpm

If we change <name>elasticsearch-${project.version}.rpm</name> to <name>elasticsearch</name> in the pom.xml, we end with:

  • Maven project:
    • elasticsearch-2.0.0-SNAPSHOT20150703094032.noarch.rpm << ok
  • Maven local repository:
    • elasticsearch-rpm-2.0.0-SNAPSHOT.rpm << wrong

It seems difficult to have a coherent naming between the built artefact and the installed artefact.

With this last change in the pom.xml, RPM works on Fedora21, CentOS6.6, CentOS7,OpenSUSE13 (I did not test tar.gz/zip/deb packages)

The jsr166e-1.1.0.jar is still missing in the packages... did you find out if it has been added in master?

Member

tlrx commented Jul 3, 2015

@dadoonet On my laptop the RPM package still has a wrong filename:

  • Maven project:
    • elasticsearch-2.0.0-SNAPSHOT.rpm-2.0.0-SNAPSHOT20150703093546.noarch.rpm
  • Maven local repository:
    • elasticsearch-rpm-2.0.0-SNAPSHOT.rpm

If we change <name>elasticsearch-${project.version}.rpm</name> to <name>elasticsearch</name> in the pom.xml, we end with:

  • Maven project:
    • elasticsearch-2.0.0-SNAPSHOT20150703094032.noarch.rpm << ok
  • Maven local repository:
    • elasticsearch-rpm-2.0.0-SNAPSHOT.rpm << wrong

It seems difficult to have a coherent naming between the built artefact and the installed artefact.

With this last change in the pom.xml, RPM works on Fedora21, CentOS6.6, CentOS7,OpenSUSE13 (I did not test tar.gz/zip/deb packages)

The jsr166e-1.1.0.jar is still missing in the packages... did you find out if it has been added in master?

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 6, 2015

Member

@tlrx Could you try to build my branch again? I don't understand why my VM does not accept it anymore and fails...

Member

dadoonet commented Jul 6, 2015

@tlrx Could you try to build my branch again? I don't understand why my VM does not accept it anymore and fails...

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 6, 2015

Member

any trick to build the RPM as part of mvn package in the root dir? Is there anything specific that fails for you or should be tested?

Member

spinscale commented Jul 6, 2015

any trick to build the RPM as part of mvn package in the root dir? Is there anything specific that fails for you or should be tested?

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 6, 2015

Member

@clintongormley @s1monw I think that this one is a requirement to split out the plugin manager into its own sub-project. Reason for this is, that the plugin-manager source has a dependency to the core, but when packaging everything together, the plugin manager jar needs to be included in the distribution. I do not see a good way doing this, when creating the different packages is part of the core building process.

Member

spinscale commented Jul 6, 2015

@clintongormley @s1monw I think that this one is a requirement to split out the plugin manager into its own sub-project. Reason for this is, that the plugin-manager source has a dependency to the core, but when packaging everything together, the plugin manager jar needs to be included in the distribution. I do not see a good way doing this, when creating the different packages is part of the core building process.

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 6, 2015

Member

@spinscale if your have rpm installed on your machine, then when running mvn package, rpm module is automatically added to the build.

See https://github.com/dadoonet/elasticsearch/blob/maven/distribution/distribution/pom.xml#L152-163

I tested /usr/bin/rpmbuild. Do you have it on your machine?

Member

dadoonet commented Jul 6, 2015

@spinscale if your have rpm installed on your machine, then when running mvn package, rpm module is automatically added to the build.

See https://github.com/dadoonet/elasticsearch/blob/maven/distribution/distribution/pom.xml#L152-163

I tested /usr/bin/rpmbuild. Do you have it on your machine?

@tlrx

This comment has been minimized.

Show comment
Hide comment
@tlrx

tlrx Jul 6, 2015

Member

@tlrx Could you try to build my branch again? I don't understand why my VM does not accept it anymore and fails...

I can build it on my laptop (commit dcc34b5e672204e86b5452ded1db8ba34e1a3d1d).

Member

tlrx commented Jul 6, 2015

@tlrx Could you try to build my branch again? I don't understand why my VM does not accept it anymore and fails...

I can build it on my laptop (commit dcc34b5e672204e86b5452ded1db8ba34e1a3d1d).

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 6, 2015

Member

@dadoonet installed rpm via brew on osx, thus it is /usr/local/bin/rpmbuild

Member

spinscale commented Jul 6, 2015

@dadoonet installed rpm via brew on osx, thus it is /usr/local/bin/rpmbuild

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 6, 2015

Member

@spinscale oh? I was not aware that we can have rpm on OSX! I spent too much time trying to work with a VM! I guess you saved me here! Thanks. So I'll will adapt the PR to check in both dirs. For now, you can edit the pom.xml file and link to your dir.

Member

dadoonet commented Jul 6, 2015

@spinscale oh? I was not aware that we can have rpm on OSX! I spent too much time trying to work with a VM! I guess you saved me here! Thanks. So I'll will adapt the PR to check in both dirs. For now, you can edit the pom.xml file and link to your dir.

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 6, 2015

Member

maybe have a property as well ${packaging.rpm.rpmbuild}, so one can configure via the commandline and no need to change files in case it is somewhere else?

Member

spinscale commented Jul 6, 2015

maybe have a property as well ${packaging.rpm.rpmbuild}, so one can configure via the commandline and no need to change files in case it is somewhere else?

@dadoonet

This comment has been minimized.

Show comment
Hide comment
Member

dadoonet commented Jul 6, 2015

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 7, 2015

Member

can you rebase this against master? It misses the changes of #11938, that removes the gpg stuff from the plugin configuration and just keeps it in the profile, that needs to be activated specifically

Member

spinscale commented Jul 7, 2015

can you rebase this against master? It misses the changes of #11938, that removes the gpg stuff from the plugin configuration and just keeps it in the profile, that needs to be activated specifically

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 7, 2015

Member

the debian package filename is elasticsearch-deb-2.0.0-SNAPSHOT.deb? that deb in the middle seems not needed. Also I got this exception when starting up (doesnt happen on master, so I suppose this is because it is on an old state?). From /var/log/elasticsearch/elasticsearch.log

[2015-07-07 13:38:36,839][ERROR][bootstrap                ] Exception
java.nio.file.AccessDeniedException: /var/lib/puppet
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:426)
        at java.nio.file.Files.newDirectoryStream(Files.java:413)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:179)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
        at java.nio.file.Files.walkFileTree(Files.java:2602)
        at java.nio.file.Files.walkFileTree(Files.java:2635)
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:136)
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:69)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:284)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)
Member

spinscale commented Jul 7, 2015

the debian package filename is elasticsearch-deb-2.0.0-SNAPSHOT.deb? that deb in the middle seems not needed. Also I got this exception when starting up (doesnt happen on master, so I suppose this is because it is on an old state?). From /var/log/elasticsearch/elasticsearch.log

[2015-07-07 13:38:36,839][ERROR][bootstrap                ] Exception
java.nio.file.AccessDeniedException: /var/lib/puppet
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:426)
        at java.nio.file.Files.newDirectoryStream(Files.java:413)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:179)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199)
        at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
        at java.nio.file.Files.walkFileTree(Files.java:2602)
        at java.nio.file.Files.walkFileTree(Files.java:2635)
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:136)
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:69)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:169)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:284)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:28)

dadoonet added some commits Jun 3, 2015

[maven] split packaging into modules
This change creates a proper `distribution` modules in which we have today packaging for:

* zip
* tar.gz
* rpm
* deb

Adding a new packaging should be adding a new module and define in its `pom.xml` any specific needs.
Update from master
TODO

remove core/src/packaging
core/src/assembly
Update after review
> I don't think that the package names should be changed (before: elasticsearch-2.0.0-SNAPSHOT.zip, in this pull request: elasticsearch-zip-2.0.0-SNAPSHOT.zip). Same thing for Deb/Tar/Zip packages.
Update after review
> /elasticsearch/core/src/packaging directory does not seem to be removed... Or maybe it's just my IDE?
Unify tar and zip modules to simplify license-checker job
Move licenses-checker to tar module

Note that we should may be reconsider having

* tar
* zip

modules and then add a QA module or whatever name we want which is responsible to check what we need to between tar and zip artifacts.
[build] Automatically execute RPM artifact build
... when `/usr/bin/rpmbuild` is available on the machine
[build] fix deb build
It was previously generating a fake elasticsearch-x.x.x.jar and was adding it twice in the debian artifact.

dadoonet added some commits Jul 1, 2015

[build] Use rpm artifact packaging
We can use `rpm` as a packaging type in maven.
It allows to automatically attach the generated rpm to the build cycle and add it as a generated artifact. So it's also published in the `.m2` and in maven central as well.
Move files from src/packaging to src/main/packaging
Also disable rpm module when not on a RPM compatible machine
@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 7, 2015

Member

@spinscale I rebased, fixed the deb naming issue and also move rpm plugin to 2.1.3 as you did in a previous change.
Will work on fixing rpm working as well on MacOSX now and other remaining stuff.

Member

dadoonet commented Jul 7, 2015

@spinscale I rebased, fixed the deb naming issue and also move rpm plugin to 2.1.3 as you did in a previous change.
Will work on fixing rpm working as well on MacOSX now and other remaining stuff.

dadoonet added some commits Jul 7, 2015

@dadoonet

This comment has been minimized.

Show comment
Hide comment
@dadoonet

dadoonet Jul 7, 2015

Member

I added 2 new changes:

  • RPM can now work also on MacOSX (thanks @spinscale for telling me I can run brew install rpm)
  • rpm final file is also copied with the right name under target/releases.

Note that when artifact SHA1 will be computed, we need to make sure this is still working as expected.

That being said, may be we should simply adapt our release tool to publish everything which has been installed in the local ~/.m2 as those are the right files... I think (but unsure) that mvn deploy actually deploy files from this ~/.m2 dir and not from target or target/releases.

@tlrx @spinscale Could you look at this please?

Also if you can share with me what exact commands you are using to test deb packages that will be easier for me to test and fix instead of pinging you all the time :).

Member

dadoonet commented Jul 7, 2015

I added 2 new changes:

  • RPM can now work also on MacOSX (thanks @spinscale for telling me I can run brew install rpm)
  • rpm final file is also copied with the right name under target/releases.

Note that when artifact SHA1 will be computed, we need to make sure this is still working as expected.

That being said, may be we should simply adapt our release tool to publish everything which has been installed in the local ~/.m2 as those are the right files... I think (but unsure) that mvn deploy actually deploy files from this ~/.m2 dir and not from target or target/releases.

@tlrx @spinscale Could you look at this please?

Also if you can share with me what exact commands you are using to test deb packages that will be easier for me to test and fix instead of pinging you all the time :).

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 13, 2015

Member

ok, two things. I needed to fix the packages to contain the working directory for start-stop-daemon and systemd. This is the diff

diff --git a/distribution/deb/src/main/packaging/init.d/elasticsearch b/distribution/deb/src/main/packaging/init.d/elasticsearch
index 0c5e91d..19198c9 100755
--- a/distribution/deb/src/main/packaging/init.d/elasticsearch
+++ b/distribution/deb/src/main/packaging/init.d/elasticsearch
@@ -173,7 +173,7 @@ case "$1" in
        fi

        # Start Daemon
-       start-stop-daemon --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
+       start-stop-daemon -d $ES_HOME --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
        return=$?
        if [ $return -eq 0 ]
        then
diff --git a/distribution/src/main/packaging/systemd/elasticsearch.service b/distribution/src/main/packaging/systemd/elasticsearch.service
index a4c2699..b857d61 100644
--- a/distribution/src/main/packaging/systemd/elasticsearch.service
+++ b/distribution/src/main/packaging/systemd/elasticsearch.service
@@ -13,6 +13,8 @@ Environment=LOG_DIR=${packaging.elasticsearch.log.dir}
 Environment=PID_DIR=${packaging.elasticsearch.pid.dir}
 EnvironmentFile=-${packaging.env.file}

+WorkingDirectory=${packaging.elasticsearch.home.dir}
+
 User=${packaging.elasticsearch.user}
 Group=${packaging.elasticsearch.group}

One bigger issue remains. The distribution/pom.xml and all the pom files below contains a SNAPSHOT parent and each sub-project contains a version of the distribution pom. Who is updating those to stable values and ensure that they do not run out of sync?

    <parent>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-parent</artifactId>
        <version>2.0.0-SNAPSHOT</version>
    </parent>

// each sub project pom.xml
    <parent>
        <groupId>org.elasticsearch.distribution</groupId>
        <artifactId>elasticsearch-distribution</artifactId>
        <version>2.0.0-SNAPSHOT</version>
    </parent>

Apart from that I tested the packages and they look good.

Member

spinscale commented Jul 13, 2015

ok, two things. I needed to fix the packages to contain the working directory for start-stop-daemon and systemd. This is the diff

diff --git a/distribution/deb/src/main/packaging/init.d/elasticsearch b/distribution/deb/src/main/packaging/init.d/elasticsearch
index 0c5e91d..19198c9 100755
--- a/distribution/deb/src/main/packaging/init.d/elasticsearch
+++ b/distribution/deb/src/main/packaging/init.d/elasticsearch
@@ -173,7 +173,7 @@ case "$1" in
        fi

        # Start Daemon
-       start-stop-daemon --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
+       start-stop-daemon -d $ES_HOME --start -b --user "$ES_USER" -c "$ES_USER" --pidfile "$PID_FILE" --exec $DAEMON -- $DAEMON_OPTS
        return=$?
        if [ $return -eq 0 ]
        then
diff --git a/distribution/src/main/packaging/systemd/elasticsearch.service b/distribution/src/main/packaging/systemd/elasticsearch.service
index a4c2699..b857d61 100644
--- a/distribution/src/main/packaging/systemd/elasticsearch.service
+++ b/distribution/src/main/packaging/systemd/elasticsearch.service
@@ -13,6 +13,8 @@ Environment=LOG_DIR=${packaging.elasticsearch.log.dir}
 Environment=PID_DIR=${packaging.elasticsearch.pid.dir}
 EnvironmentFile=-${packaging.env.file}

+WorkingDirectory=${packaging.elasticsearch.home.dir}
+
 User=${packaging.elasticsearch.user}
 Group=${packaging.elasticsearch.group}

One bigger issue remains. The distribution/pom.xml and all the pom files below contains a SNAPSHOT parent and each sub-project contains a version of the distribution pom. Who is updating those to stable values and ensure that they do not run out of sync?

    <parent>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-parent</artifactId>
        <version>2.0.0-SNAPSHOT</version>
    </parent>

// each sub project pom.xml
    <parent>
        <groupId>org.elasticsearch.distribution</groupId>
        <artifactId>elasticsearch-distribution</artifactId>
        <version>2.0.0-SNAPSHOT</version>
    </parent>

Apart from that I tested the packages and they look good.

@spinscale

This comment has been minimized.

Show comment
Hide comment
@spinscale

spinscale Jul 27, 2015

Member

obsoleted by #12286

Member

spinscale commented Jul 27, 2015

obsoleted by #12286

@spinscale spinscale closed this Jul 27, 2015

@dadoonet dadoonet deleted the dadoonet:maven/distribution branch Aug 18, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment