Can't install elasticsearch plugins

[CapSel]

Issue description

There is an error in install stage when configuring plugins for elastic search. It is present in 18.09 channel and on master branch of nixpkgs.

Steps to reproduce

1. Put in /etc/nixos/configuration.nix

    services.elasticsearch = {
      package = pkgs.elasticsearch-oss;
      enable = true;
      listenAddress = "127.0.0.1";
      plugins = with pkgs.elasticsearchPlugins; [ search_guard ];
    };

2. run nixos-rebuild switch --upgrade and watch the error

# nixos-rebuild switch --upgrade
unpacking channels...
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/1japx4r2c62bg73lcliqym2cdv6rs1jy-elastic-search-guard-6.5.1-23.2.drv
  /nix/store/h6ix04i05f4phq1iw5pgfv85pqz2flp4-elasticsearch-plugins.drv
  /nix/store/06fappd5hvca2pnwacp44c79dmiqnq02-unit-script-elasticsearch-pre-start.drv
  /nix/store/n21q7p1bd0c4d06rgqc92c0zpnx5xyjl-unit-elasticsearch.service.drv
  /nix/store/5yqg22vjg8kab686lqh89975pmvnc4bg-system-units.drv
  /nix/store/0dfi2vk99dqnrsnj5rdw2s982c1fzmz7-etc.drv
  /nix/store/svljia18z1s7g4dkiyr83ydlkvcjg11j-nixos-system-bonsai-18.09.2292.4c0be408155.drv
building '/nix/store/1japx4r2c62bg73lcliqym2cdv6rs1jy-elastic-search-guard-6.5.1-23.2.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
Error: Could not find or load main class org.elasticsearch.tools.java_version_checker.JavaVersionChecker
builder for '/nix/store/1japx4r2c62bg73lcliqym2cdv6rs1jy-elastic-search-guard-6.5.1-23.2.drv' failed with exit code 1
cannot build derivation '/nix/store/h6ix04i05f4phq1iw5pgfv85pqz2flp4-elasticsearch-plugins.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/06fappd5hvca2pnwacp44c79dmiqnq02-unit-script-elasticsearch-pre-start.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n21q7p1bd0c4d06rgqc92c0zpnx5xyjl-unit-elasticsearch.service.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5yqg22vjg8kab686lqh89975pmvnc4bg-system-units.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/0dfi2vk99dqnrsnj5rdw2s982c1fzmz7-etc.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/svljia18z1s7g4dkiyr83ydlkvcjg11j-nixos-system-bonsai-18.09.2292.4c0be408155.drv': 1 dependencies couldn't be built
error: build of '/nix/store/svljia18z1s7g4dkiyr83ydlkvcjg11j-nixos-system-bonsai-18.09.2292.4c0be408155.drv' failed

3. Or from master:

[root@bonsai:~/nixpkgs]# git pull --rebase
Already up to date.
Current branch master is up to date.

[root@bonsai:~/nixpkgs]# git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

[root@bonsai:~/nixpkgs]# nix-build /root/nixpkgs/ -A pkgs.elasticsearchPlugins.search_guard
these derivations will be built:
  /nix/store/vis8n7i9b9w8pxd5wzqvdmzsf3b2by9d-elastic-search-guard-6.5.1-23.2.drv
building '/nix/store/vis8n7i9b9w8pxd5wzqvdmzsf3b2by9d-elastic-search-guard-6.5.1-23.2.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
no Makefile, doing nothing
installing
Error: Could not find or load main class org.elasticsearch.tools.java_version_checker.JavaVersionChecker
builder for '/nix/store/vis8n7i9b9w8pxd5wzqvdmzsf3b2by9d-elastic-search-guard-6.5.1-23.2.drv' failed with exit code 1
error: build of '/nix/store/vis8n7i9b9w8pxd5wzqvdmzsf3b2by9d-elastic-search-guard-6.5.1-23.2.drv' failed

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

[root@bonsai:~]# nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/ial8a56ys2rswfq4j2abm3cl99li5b6p-bash-interactive-4.4-p23-dev
copying path '/nix/store/ial8a56ys2rswfq4j2abm3cl99li5b6p-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
error: restoring parent mount namespace: Invalid argument

[root@bonsai:~]# nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.104, NixOS, 18.09.2292.4c0be408155 (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2`
 - channels(root): `"nixos-18.09.2292.4c0be408155, nixpkgs-19.03pre161900.61c3169a0e1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[betaboon]

i have been playing around with this a little.

What i have found so far:
it can made to work by symlinking elasticsearch-oss when building the package.

this has to be done here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/search/elasticsearch/plugins.nix#L9

resulting in:

    installPhase ? ''
      mkdir -p $out/config
      mkdir -p $out/plugins
      ln -s ${elasticsearch-oss}/lib $out/lib
      ES_HOME=$out ${elasticsearch-oss}/bin/elasticsearch-plugin install --batch -v file://$src
      rm $out/lib
    ''

BUT:
the elasticsearch-service-modules symlinks the plugins into ${cfg.dataDir}/plugins here

this results in elasticsearch trying to load the plugins but failing with access denied ("java.io.FilePermission" just as mentioned here in the case of symlinking the elasticsearch-config.

this can be worked around by copying the plugins instead of symlinking here.
resulting in:

        mkdir -p ${cfg.dataDir}/plugins
        for plugin in  ${concatStringsSep " " cfg.plugins}; do
          cp -r $plugin/plugins/* ${cfg.dataDir}/plugins
        done

but this kind of seems crappy.

the root cause might be this:
https://www.elastic.co/guide/en/elasticsearch/reference/2.1/breaking_20_plugin_and_packaging_changes.html#_symbolic_links_and_paths

[betaboon]

changing the module to

  esPlugins = pkgs.stdenv.mkDerivation {
    name = "elasticsearch-plugins";
    buildCommand = ''
      mkdir -p $out/plugins
      for plugin in  ${concatStringsSep " " cfg.plugins}; do
        cp -r $plugin/plugins/* $out/plugins
      done
    '';
  };

seems to work, which lets me believe that the aforementioned link is relevant here.

A configured path may itself be a symbolic link, but no symlinks under that path will be followed.

[CapSel]

Any news? It's almost a year.

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.