Cannot remove plugin from docker container #24231

tback · 2017-04-21T06:51:50Z

It seems uninstalling plugins from the official docker image is broken.

My colleagues were unable to reproduce this error on debian stretch with btrfs.

I'm opening this issue in response to @jasontedor #23548 (comment)

Elasticsearch version:
5.3.1

Plugins installed: [x-pack]

JVM version:

docker run -ti --rm --name es docker.elastic.co/elasticsearch/elasticsearch:5.3.1 /usr/lib/jvm/java-1.8-openjdk/bin/java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (IcedTea 3.3.0) (Alpine 8.121.13-r0)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version:
macos sierra 10.12.3 (16D32)
Docker Version 17.03.1-ce-mac5 (16048, b18e2a50cc)

Description of the problem including expected versus actual behavior:
I expect to successfully remove x-pack like any other plugin from the official docker image. I get an error instead.
I'd much rather get an official image without x-pack altogether, but thats a business decision I guess. Making x-pack hard to uninstall will drive me to community images for sure.

Steps to reproduce:

Create a Dockerfile


RUN eval ${ES_JAVA_OPTS:-} && \
    elasticsearch-plugin remove --silent x-pack

Try to build it
docker build .
Trace

Sending build context to Docker daemon 2.048 kB
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1
5.3.1: Pulling from elasticsearch/elasticsearch
ec37562cf8fa: Pull complete
3f5b02e577c3: Pull complete
e4c7c0eb9289: Pull complete
e8008e68933d: Pull complete
1cfca4b45f64: Pull complete
77148a7a523b: Pull complete
c93eabd5eff3: Pull complete
d82d8a15579f: Pull complete
0f2b8d9540a0: Pull complete
5c9aa0d9a460: Pull complete
Digest: sha256:94ebca9d6316902596ceef6934161be39cca04885fcf07c8ab3feb04c02b4933
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:5.3.1
 ---> 94062e7e850e
Step 2/2 : RUN eval ${ES_JAVA_OPTS:-} &&     elasticsearch-plugin remove --silent x-pack
 ---> Running in c7a27bb18817
Exception in thread "main" java.nio.file.DirectoryNotEmptyException: /usr/share/elasticsearch/plugins/x-pack
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:91)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:56)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
	at org.elasticsearch.cli.Command.main(Command.java:88)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c eval ${ES_JAVA_OPTS:-} &&     elasticsearch-plugin remove --silent x-pack' returned a non-zero code: 1

The text was updated successfully, but these errors were encountered:

tback · 2017-04-21T07:14:42Z

This is also easily reproduced on the vagrant base box debian/jessie64.

I suspect this is related to aufs.

Elasticsearch version:
5.3.1

Plugins installed: [x-pack]

JVM version:
(Same base image, same jvm version)

openjdk version "1.8.0_121"
OpenJDK Runtime Environment (IcedTea 3.3.0) (Alpine 8.121.13-r0)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

OS version:
debian jessie 64bit

uname -a
Linux jessie 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux
docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:07:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Steps to reproduce

vagrant init debian/jessie64
vagrant up
vagrant ssh

Then, on the machine
install docker

sudo apt-get install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg2 \
     software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce
sudo adduser vagrant docker

logout && login again (so the new group becomes effective

Create Dockerfile with same contents as above:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1

RUN eval ${ES_JAVA_OPTS:-} && \
    elasticsearch-plugin remove --silent x-pack

Run the docker build

vagrant@jessie:~$ docker build .

And get the trace:

Sending build context to Docker daemon 13.31 kB
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.1
5.3.1: Pulling from elasticsearch/elasticsearch
ec37562cf8fa: Pull complete
3f5b02e577c3: Pull complete
e4c7c0eb9289: Pull complete
e8008e68933d: Pull complete
1cfca4b45f64: Pull complete
77148a7a523b: Pull complete
c93eabd5eff3: Pull complete
d82d8a15579f: Pull complete
0f2b8d9540a0: Pull complete
5c9aa0d9a460: Pull complete
Digest: sha256:94ebca9d6316902596ceef6934161be39cca04885fcf07c8ab3feb04c02b4933
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:5.3.1
 ---> 94062e7e850e
Step 2/2 : RUN eval ${ES_JAVA_OPTS:-} &&     elasticsearch-plugin remove --silent x-pack
 ---> Running in b6e5800a9b42
Exception in thread "main" java.nio.file.DirectoryNotEmptyException: /usr/share/elasticsearch/plugins/x-pack
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:91)
	at org.elasticsearch.plugins.RemovePluginCommand.execute(RemovePluginCommand.java:56)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
	at org.elasticsearch.cli.Command.main(Command.java:88)
	at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c eval ${ES_JAVA_OPTS:-} &&     elasticsearch-plugin remove --silent x-pack' returned a non-zero code: 1

jasontedor · 2017-04-21T16:24:33Z

Thanks for the report @tback. I've found the source of the issue; it's a combination of renaming not working on the lower layer of a union filesystem and a bug in the JDK. Briefly the JDK tries to do the following:

if it's an atomic move, try to rename the source to target and if that fails with EXDEV, throw atomic move not supported
if it's not an atomic move, try to rename the source except this time do not fail if EXDEV occurred
when the previous rename fails, copy source to target and now try to rmdir the source; the bug is that rmdir requires the source to be empty but the JDK never cleans up the source so this will fail 100% of the time

I'd much rather get an official image without x-pack altogether, but thats a business decision I guess. Making x-pack hard to uninstall will drive me to community images for sure.

Your feedback here is very much appreciated. I want you to know that we are not intentionally making it difficult to uninstall X-Pack, we will prepare a fix for this for 5.4.0.

jasontedor · 2017-04-21T18:33:57Z

We will prepare a fix for this for 5.4.0.

I've opened #24252. We are going to try to get this into a 5.3.2 release (do not take that as a guarantee). I've built a new Docker image locally based off this change and the issue no longer reproduces.

dliappis · 2017-04-23T14:12:06Z

Thanks for taking care of this @jasontedor and reporting it @tback. I've taken a note to add an acceptance test for removing plugins in our docker image building acceptance tests. This will most likely land in 5.4.0 where we have introduced a more flexible acceptance test framework.

Plugin management code in Elasticsearch should work in union filesystems as implemented by various Docker Storage Drivers. Test that the x-pack plugin can be uninstalled correctly as part of the acceptance tests. Relates: elastic/elasticsearch#24231

jasontedor mentioned this issue Apr 21, 2017

Use a marker file when removing a plugin #24252

Merged

jasontedor closed this as completed in #24252 Apr 21, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot remove plugin from docker container #24231

Cannot remove plugin from docker container #24231

tback commented Apr 21, 2017 •

edited

tback commented Apr 21, 2017 •

edited

jasontedor commented Apr 21, 2017 •

edited

jasontedor commented Apr 21, 2017

dliappis commented Apr 23, 2017

Cannot remove plugin from docker container #24231

Cannot remove plugin from docker container #24231

Comments

tback commented Apr 21, 2017 • edited

tback commented Apr 21, 2017 • edited

jasontedor commented Apr 21, 2017 • edited

jasontedor commented Apr 21, 2017

dliappis commented Apr 23, 2017

tback commented Apr 21, 2017 •

edited

tback commented Apr 21, 2017 •

edited

jasontedor commented Apr 21, 2017 •

edited