elasticsearch: use 6.x as default version, remove unsupported releases #44389

Mic92 · 2018-08-03T09:03:50Z

Motivation for this change

I had to drop some unsupported plugins as well. Alternative to #44339 The new/updated plugins build, but I have not tested them yet.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

basvandijk · 2018-08-03T09:19:13Z

pkgs/top-level/all-packages.nix

@@ -2273,12 +2273,10 @@ with pkgs;
  elk6Version = "6.3.2";

  elasticsearch = callPackage ../servers/search/elasticsearch { };
-  elasticsearch2 = callPackage ../servers/search/elasticsearch/2.x.nix { };
-  elasticsearch5 = callPackage ../servers/search/elasticsearch/5.x.nix { };
-  elasticsearch6 = callPackage ../servers/search/elasticsearch/6.x.nix { };


What about keeping the elasticsearch6 and elasticsearch6-oss attributes and setting the aliases elasticsearch = elasticsearch6 and elasticsearch-oss = elasticsearch6-oss. That causes less breakage. Note that we do the same for postgresql, linuxPackages, etc..

We should also add an entry to the release notes warning that elasticsearch has been upgraded to version 6.

We should probably also mention that ~~elasticsearch~~ (EDIT: the ELK stack) now has an unfree license because it contains X-pack but that you can use the free elasticsearch-oss, kibana-oss and logstash-oss instead.

basvandijk · 2018-08-03T09:23:50Z

nixos/modules/services/search/elasticsearch.nix

@@ -58,8 +58,8 @@ in {

    package = mkOption {
      description = "Elasticsearch package to use.";
-      default = pkgs.elasticsearch2;
-      defaultText = "pkgs.elasticsearch2";
+      default = pkgs.elasticsearch;


Note that we can now simplify the module by replacing all if es5 then ... else ... expressions with their then branches. (Note that es5 means: version 5 or higher.)

basvandijk · 2018-08-03T09:41:24Z

This is a good change! However, we should handle the other components in the ELK stack as well:

kibana is at 4.6.5 which is also EOL. So we should set: kibana = kibana6
logstash is at 2.4.0 which is also EOL. So we should set: logstash = logstash6
Then we have the beats: filebeat, heartbeat, metricbeat and packetbeat which are currently pointing to version 5. I propose we set filebeat = filebeat6, heartbeat = heartbeat6, metricbeat = metricbeat6 and packetbeat = packetbeat6. And add the new attributes: filebeat5, heartbeat5, metricbeat5 and packetbeat5 and point them to version 5.

River plugins are no longer supported in elasticsearch.

Deprecated by upstream, search_guard was recommended instead.

deprecated by upstream, https://github.com/lmenezes/cerebro is recommend instead (but not packaged in nixpkgs)

Mic92 · 2018-08-03T10:54:09Z

@GrahamcOfBorg test nixos.tests.elk.ELK-6

GrahamcOfBorg · 2018-08-03T10:54:37Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.nixos.tests.elk.ELK-6

Partial log (click to expand)

Cannot nix-instantiate `tests.nixos.tests.elk.ELK-6' because:
error: attribute 'nixos' in selection path 'tests.nixos.tests.elk.ELK-6' not found

GrahamcOfBorg · 2018-08-03T10:54:43Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: tests.nixos.tests.elk.ELK-6

Partial log (click to expand)

Cannot nix-instantiate `tests.nixos.tests.elk.ELK-6' because:
error: attribute 'nixos' in selection path 'tests.nixos.tests.elk.ELK-6' not found

Mic92 · 2018-08-03T10:55:44Z

@GrahamcOfBorg test elk.ELK-6

GrahamcOfBorg · 2018-08-03T10:56:13Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.elk.ELK-6

Partial log (click to expand)

Cannot nix-instantiate `tests.elk.ELK-6' because:
error: attribute 'ELK-6' in selection path 'tests.elk.ELK-6' not found

GrahamcOfBorg · 2018-08-03T11:36:38Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.elk.ELK-5

Partial log (click to expand)

Cannot nix-instantiate `tests.elk.ELK-5' because:
error: attribute 'ELK-5' in selection path 'tests.elk.ELK-5' not found

GrahamcOfBorg · 2018-08-03T11:53:19Z

Success on x86_64-linux (full log)

Attempted: tests.elk.ELK-5

Partial log (click to expand)

one: exit status 1
syncing
one: running command: sync
one: exit status 0
test script finished in 79.98s
cleaning up
killing one (pid 597)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/h9isl5ilmffz9357hjkj0m8fp40kd30j-vm-test-run-ELK-5

basvandijk · 2018-08-03T13:43:11Z

@Mic92 In master...LumiGuide:es6 I've added a commit that updates the 18.09 release notes. You may want to pull that into your PR.

Mic92 · 2018-08-03T13:55:29Z

@basvandijk you can also push it by using: git push git@github.com/Mic92/nixpkgs HEAD:es6
as your are member of nixpkgs.

Mic92 · 2018-08-03T14:07:28Z

pushed the changelog as well

Mic92 · 2018-08-03T14:09:55Z

nixos/doc/manual/release-notes/rl-1809.xml

+     Since X-Pack is licensed under the
+     <link xlink:href="https://github.com/elastic/elasticsearch/blob/master/licenses/ELASTIC-LICENSE.txt">Elastic License</link>
+     the ELK packages now have an unfree license. To use them you need to specify
+     <literal>allowUnfree = true;</literal> in your nixpkgs configuration.


Did you mean?

nixpkgs.config.allowUnfree = true;

Yes but I didn't want to assume NixOS. Although people are probably not using the ELK stack outside of NixOS...

basvandijk · 2018-08-23T21:53:38Z

@Mic92 I fixed some conflicts with master. I think this is ready to be merged right?

basvandijk · 2018-08-23T21:55:27Z

@GrahamcOfBorg test elk.ELK-5 elk.ELK-6

GrahamcOfBorg · 2018-08-23T21:55:52Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.elk.ELK-5, tests.elk.ELK-6

Partial log (click to expand)

Cannot nix-instantiate `tests.elk.ELK-5' because:
error: attribute 'ELK-5' in selection path 'tests.elk.ELK-5' not found

Cannot nix-instantiate `tests.elk.ELK-6' because:
error: attribute 'ELK-6' in selection path 'tests.elk.ELK-6' not found

GrahamcOfBorg · 2018-08-23T22:03:17Z

Success on x86_64-linux (full log)

Attempted: tests.elk.ELK-5, tests.elk.ELK-6

Partial log (click to expand)

syncing
one: running command: sync
one: exit status 0
test script finished in 159.73s
cleaning up
killing one (pid 609)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/8mizvwwigf9g8wlrsf4ikzb0j96bb1f3-vm-test-run-ELK-5
/nix/store/4qpnb1k8ssm6mdrhh0wfki6mysinw6p5-vm-test-run-ELK-6

GrahamcOfBorg · 2018-08-23T22:05:04Z

Success on x86_64-darwin (full log)

Attempted: elasticsearchPlugins.discovery-ec2, elasticsearchPlugins.elasticsearch_analysis_lemmagen

The following builds were skipped because they don't evaluate on x86_64-darwin: elasticsearch, elasticsearch6, kibana, logstash

Partial log (click to expand)

* java.lang.RuntimePermission getClassLoader
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed discovery-ec2
post-installation fixup
/nix/store/5im29qx9rni3w8l3j8i2yr22sin55kxv-cctools-binutils-darwin/bin/strip is /nix/store/5im29qx9rni3w8l3j8i2yr22sin55kxv-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/8zq6jzd4jglrfaahli2a4yxfl66p7wy2-elasticsearch-discovery-ec2-18.09pre-git
/nix/store/8zq6jzd4jglrfaahli2a4yxfl66p7wy2-elasticsearch-discovery-ec2-18.09pre-git
/nix/store/bp0lbinr2qc62gn9g0m8xgfg0f8mj6zd-elasticsearch-analysis-lemmagen-6.3.2

GrahamcOfBorg · 2018-08-23T22:05:15Z

Success on x86_64-linux (full log)

Attempted: elasticsearchPlugins.discovery-ec2, elasticsearchPlugins.elasticsearch_analysis_lemmagen

The following builds were skipped because they don't evaluate on x86_64-linux: elasticsearch, elasticsearch6, kibana, logstash

Partial log (click to expand)

 * Classname: org.elasticsearch.plugin.analysis.lemmagen.AnalysisLemmagenPlugin
plugin has a policy file with no additional permissions
-> Installed elasticsearch-analysis-lemmagen
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7ll1sznzpz1m9wdd5s2dg5j9z9d3gmnp-elasticsearch-analysis-lemmagen-6.3.2
/nix/store/vp40my263xk32f07788srrgl59rycsna-binutils-2.30/bin/strip is /nix/store/vp40my263xk32f07788srrgl59rycsna-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/7ll1sznzpz1m9wdd5s2dg5j9z9d3gmnp-elasticsearch-analysis-lemmagen-6.3.2
checking for references to /build in /nix/store/7ll1sznzpz1m9wdd5s2dg5j9z9d3gmnp-elasticsearch-analysis-lemmagen-6.3.2...
/nix/store/9f6f1kcdbnbjxsxm9afqxnaa302dilgw-elasticsearch-discovery-ec2-18.09pre-git
/nix/store/7ll1sznzpz1m9wdd5s2dg5j9z9d3gmnp-elasticsearch-analysis-lemmagen-6.3.2

GrahamcOfBorg · 2018-08-23T22:05:16Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: elasticsearch, elasticsearch6, elasticsearchPlugins.discovery-ec2, elasticsearchPlugins.elasticsearch_analysis_lemmagen, kibana, logstash

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

basvandijk · 2018-08-23T23:26:28Z

Note I'm also working on the ELK 6.3.2 -> 6.4.0 upgrade. The tests pass but the plugins: elasticsearch_analysis_lemmagen and search_guard haven't upgraded to 6.4 yet so they fail to install.

Mic92 · 2018-08-24T13:07:40Z

@basvandijk feel free to push on this branch, I am currently not working on this.

basvandijk · 2018-08-24T13:19:49Z

@Mic92 I think I will just merge it this evening.

elasticsearch: drop 1.7.2 (not supported upstream)

f644675

Mic92 mentioned this pull request Aug 3, 2018

elasticsearch6: refactor plugins #44339

Closed

9 tasks

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) labels Aug 3, 2018

basvandijk suggested changes Aug 3, 2018

View reviewed changes

basvandijk reviewed Aug 3, 2018

View reviewed changes

Mic92 force-pushed the es6 branch from f849f45 to 757f82c Compare August 3, 2018 10:06

Mic92 added the 2.status: work-in-progress label Aug 3, 2018

Mic92 and others added 10 commits August 3, 2018 11:50

elasticsearch6: new default version

bedf471

elasticsearch2: drop unsupported version

1b2b62b

elasticsearchPlugins: updated for elasticsearch6

cba348c

elasticsearchPlugins.{river_jdbc,river_twitter}: remove

fa26b59

River plugins are no longer supported in elasticsearch.

elasticsearchPlugins.elasticsearch_http_basic: remove

5d3b75a

Deprecated by upstream, search_guard was recommended instead.

elasticsearchPlugins.elasticsearch_kopf: remove

0bbf90e

deprecated by upstream, https://github.com/lmenezes/cerebro is recommend instead (but not packaged in nixpkgs)

elasticsearchPlugins.elasticsearch_analysis_lemmagen: 0.1 -> 6.3.2

dab5fd5

elasticsarchPlugins.search_guard: 0.5 -> 6.3.2-22.3

19fb8c7

elasticsearchPlugins.discovery-ec2: init at 6.3.2

806ab02

nixos/elasticsearch: remove es5 version guard

cd62790

Mic92 force-pushed the es6 branch from 91b76ef to 41e8098 Compare August 3, 2018 10:51

Mic92 force-pushed the es6 branch from 41e8098 to badebba Compare August 3, 2018 11:01

Mic92 added 3 commits August 3, 2018 12:06

logstash: update default version to v6

13db07a

kibana: update default version to v6

c3055ab

beats: upgrade default from v5 to v6

d502532

Mic92 force-pushed the es6 branch from badebba to d502532 Compare August 3, 2018 11:07

GrahamcOfBorg added 8.has: clean-up 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Aug 3, 2018

Mic92 and others added 2 commits August 3, 2018 15:03

nixos/elasticsearch: fix evaluation on systems without unfree set

c9b41aa

nixos/doc: mention the ELK changes in the 18.09 release notes

3f889f3

Mic92 commented Aug 3, 2018

View reviewed changes

GrahamcOfBorg added 8.has: changelog 8.has: documentation labels Aug 3, 2018

basvandijk approved these changes Aug 3, 2018

View reviewed changes

Mic92 removed the 2.status: work-in-progress label Aug 3, 2018

Merge branch 'master' into es6

551fec4

basvandijk merged commit 7d04961 into NixOS:master Aug 25, 2018

Mic92 deleted the es6 branch August 25, 2018 19:53

flokli mentioned this pull request Apr 8, 2019

[WIP] pythonPackages.elasticsearch[-dsl] 5.x and 6.x #41201

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elasticsearch: use 6.x as default version, remove unsupported releases #44389

elasticsearch: use 6.x as default version, remove unsupported releases #44389

Mic92 commented Aug 3, 2018 •

edited

basvandijk Aug 3, 2018

basvandijk Aug 3, 2018 •

edited

basvandijk Aug 3, 2018

Mic92 Aug 3, 2018

basvandijk commented Aug 3, 2018

Mic92 commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

Mic92 commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

basvandijk commented Aug 3, 2018

Mic92 commented Aug 3, 2018

Mic92 commented Aug 3, 2018

Mic92 Aug 3, 2018

basvandijk Aug 3, 2018

basvandijk commented Aug 23, 2018

basvandijk commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

basvandijk commented Aug 23, 2018

Mic92 commented Aug 24, 2018

basvandijk commented Aug 24, 2018

elasticsearch: use 6.x as default version, remove unsupported releases #44389

elasticsearch: use 6.x as default version, remove unsupported releases #44389

Conversation

Mic92 commented Aug 3, 2018 • edited

Motivation for this change

Things done

basvandijk Aug 3, 2018

Choose a reason for hiding this comment

basvandijk Aug 3, 2018 • edited

Choose a reason for hiding this comment

basvandijk Aug 3, 2018

Choose a reason for hiding this comment

Mic92 Aug 3, 2018

Choose a reason for hiding this comment

basvandijk commented Aug 3, 2018

Mic92 commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

Mic92 commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

GrahamcOfBorg commented Aug 3, 2018

basvandijk commented Aug 3, 2018

Mic92 commented Aug 3, 2018

Mic92 commented Aug 3, 2018

Mic92 Aug 3, 2018

Choose a reason for hiding this comment

basvandijk Aug 3, 2018

Choose a reason for hiding this comment

basvandijk commented Aug 23, 2018

basvandijk commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

GrahamcOfBorg commented Aug 23, 2018

basvandijk commented Aug 23, 2018

Mic92 commented Aug 24, 2018

basvandijk commented Aug 24, 2018

Mic92 commented Aug 3, 2018 •

edited

basvandijk Aug 3, 2018 •

edited