Remove broken exec build target, replace with something better.

[rmuir]

Currently this target is "yet another way" to run elasticsearch, which we can't maintain since its an untested configuration. Besides the fact that even if we fixed it, it would break all the time since its untested, It also has the problem that it doesnt ensure its running on the latest source code, doesn't configure any scratch space properly, won't work with securitymanager, list goes on.

Instead, mvn package -Drun -DskipTests will run packaging, and then startup bin/elasticsearch(.bat) .. like integration tests, but in foreground).

It also enables debugger socket on port 8000, for people that like IDE debuggers and not system.out.println.

Its a little slower to execute because of all the shading/RPM/DEB building going on in package but that is just what it is right now until that stuff is moved out.

Logic is kind of in a wierd place, but thats also the point: its using our integration test logic that is in jenkins, so it is less likely to break.

Looks like this:

...
[INFO] Attaching created debian package /home/rmuir/workspace/elasticsearch/core/target/releases/elasticsearch-2.0.0-beta1-SNAPSHOT.deb
[INFO] 
[INFO] --- maven-antrun-plugin:1.8:run (execute) @ elasticsearch ---
[INFO] Executing tasks
main:
stop-external-cluster:
start-foreground:
   [delete] Deleting directory /home/rmuir/workspace/elasticsearch/core/target/integ-tests
    [unzip] Expanding: /home/rmuir/workspace/elasticsearch/core/target/releases/elasticsearch-2.0.0-beta1-SNAPSHOT.zip into /home/rmuir/workspace/elasticsearch/core/target/integ-tests
     [exec] Listening for transport dt_socket at address: 8000
     [exec] [2015-07-17 00:16:12,421][INFO ][node                     ] [smoke_tester] version[2.0.0-beta1-SNAPSHOT], pid[22803], build[48bc49c/2015-07-17T04:15:40Z]
     [exec] [2015-07-17 00:16:12,421][INFO ][node                     ] [smoke_tester] initializing ...
     [exec] [2015-07-17 00:16:12,423][INFO ][plugins                  ] [smoke_tester] loaded [], sites []
     [exec] [2015-07-17 00:16:12,493][INFO ][env                      ] [smoke_tester] using [1] data paths, mounts [[/ (/dev/disk/by-uuid/1b3a9e9c-3d5f-412e-be52-a7e64433e25d)]], net usable_space [205.5gb], net total_space [229.1gb], spins? [no], types [ext4]
     [exec] [2015-07-17 00:16:13,631][INFO ][node                     ] [smoke_tester] initialized
     [exec] [2015-07-17 00:16:13,631][INFO ][node                     ] [smoke_tester] starting ...
     [exec] [2015-07-17 00:16:13,710][INFO ][transport                ] [smoke_tester] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
     [exec] [2015-07-17 00:16:13,716][INFO ][discovery                ] [smoke_tester] prepare_release/yEiAMd9WTMmUMYBn1EUFtQ
     [exec] [2015-07-17 00:16:16,736][INFO ][cluster.service          ] [smoke_tester] new_master [smoke_tester][yEiAMd9WTMmUMYBn1EUFtQ][beast][inet[/127.0.0.1:9300]], reason: zen-disco-join(elected_as_master, [0] joins received)
     [exec] [2015-07-17 00:16:16,771][INFO ][http                     ] [smoke_tester] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
     [exec] [2015-07-17 00:16:16,772][INFO ][node                     ] [smoke_tester] started
     [exec] [2015-07-17 00:16:16,795][INFO ][gateway                  ] [smoke_tester] recovered [0] indices into cluster_state
     ....

[dakrone]

LGTM, two questions:

• is it possible to hook it up to compilation or the clean target automatically? I accidentally ran it without a clean in front and got: https://gist.github.com/dakrone/3c3347e263cf2e474dc2
• I notice it's no longer possible to run multiple instances of ES out of the same checkout with this (I used to run mvn exec:exec 3 times in a row on a checkout to run 3 nodes), is that intended? (no worries either way, I can look at it myself if you don't want to)

Regardless, thanks for fixing the exec target, I will use this all the time!

[rmuir]

is it possible to hook it up to compilation or the clean target automatically? I accidentally ran it without a clean in front and got: https://gist.github.com/dakrone/3c3347e263cf2e474dc2

Its hooked into compilation. You got something else going on...

I notice it's no longer possible to run multiple instances of ES out of the same checkout with this (I used to run mvn exec:exec 3 times in a row on a checkout to run 3 nodes), is that intended? (no worries either way, I can look at it myself if you don't want to)

not easily. currently this shares the integration test directory. I am getting really worried here honestly. I think if you want to run 3 nodes you should not use targets like this but run elasticsearch properly.

We gotta keep it simple...

[jpountz]

LGTM

[rmuir]

I am hostaging this pull request until we get consensus on the multiple nodes stuff. This is just not a direction i think we should take.

[dakrone]

I am hostaging this pull request until we get consensus on the multiple nodes stuff. This is just not a direction i think we should take.

My +1 is totally not contingent on either of the questions. I'm +1 to push as-is. I understand it's not something we should support, I just wondered if it was a byproduct.

[dakrone]

Its hooked into compilation. You got something else going on...

This is reproducible for me:

git checkout c2c8956
mvn clean compile test-compile
git checkout pr/12307
mvn package -Drun -DskipTests

Then - http://p.draines.com/1437144342682a61b674a.txt

(there's nothing special about c2c8956, I just picked an older commit from a week ago).

I guess I expect Maven to see that files have been changed and compile/clean accordingly? Is this an incorrect expectation? (I don't know Maven very well at all)

Again, not anything that should prevent this PR from being merged, I'm just trying to understand Maven's behavior.

[rmuir]

I guess I expect Maven to see that files have been changed and compile/clean accordingly? Is this an incorrect expectation? (I don't know Maven very well at all

This pr depends on mvn package. In fact it is just running mvn package but with flags that say "ok also start a server afterwards".

so that problem you have, I mean it is a problem, but its unrelated to this PR and an existing problem in mvn package?

The .zip file mvn package builds is broken in some cases if the workspace is dirty. If we were to release that, all users would get that exception.

Lets open a side issue for that one. It means that maven package isn't working well if the workspace is dirty.

[dakrone]

Lets open a side issue for that one. It means that maven package isn't working well if the workspace is dirty.

Sounds good, I'll open an issue for this.

[jpountz]

The issue might be in Maven itself? https://cwiki.apache.org/confluence/display/MAVEN/Incremental+Builds

[rmuir]

I understand it's not something we should support, I just wondered if it was a byproduct.

I think its something we should not try to do, until the integration tests support multiple nodes and have jobs running in jenkins that do that. Indeed this is something we probably need, to hook in backwards compatibility tests nicely and other things. But we don't have it today.

The main point of this PR is to give a target that doesn't break every day for peoples IDEs/dev environments. The whole trick here to achieve this is to lean on stuff that is tested by jenkins. I think this is really important and if we start doing fancy stuff just for the dev environment only, then we are right back where we are today, no better than having the current broken mvn exec:exec.

So I'd rather tackle the tests first... and in general with all of this, we need to walk before we can run.

[dakrone]

That sounds very reasonable, thanks for the explanation!