Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to created node environment #21

Closed
Madhu1512 opened this issue Nov 27, 2016 · 21 comments

Comments

Projects
None yet
7 participants
@Madhu1512
Copy link

commented Nov 27, 2016

Bug description

I am trying to map my local instance volume as data drive with docker image. When i run the docker image its throwing java.nio.file.AccessDeniedException

Docker command:

docker run --rm -p 9200:9200 -v /elasticsearchdb/:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.0.1

ERROR from logs:

[2016-11-27T16:51:28,964][INFO ][o.e.n.Node ] [] initializing ...
[2016-11-27T16:51:28,992][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to created node environment
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:116) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-5.0.1.jar:5.0.1]
Caused by: java.lang.IllegalStateException: Failed to created node environment
at org.elasticsearch.node.Node.(Node.java:243) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.node.Node.(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.1.jar:5.0.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_92-internal]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_92-internal]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_92-internal]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.node.Node.(Node.java:240) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.node.Node.(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.1.jar:5.0.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.1.jar:5.0.1]
... 6 more

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2016

@Madhu1512 You need to ensure that the docker container can write to this directory. The image runs Elasticsearch using the user elasticsearch (uid: 1000, gid: 1000) so please ensure the directory you are trying to mount is writable by this user. This is described in: https://discuss.elastic.co/t/elastic-elasticsearch-docker-not-assigning-permissions-to-data-directory-on-run/65812/4

@Madhu1512

This comment has been minimized.

Copy link
Author

commented Nov 27, 2016

Thanks @dliappis. I changed the permission on mount dir to 1000:1000 and i was able to write the data now and no longer seeing exceptions.

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2016

Awesome!

@Levino

This comment has been minimized.

Copy link

commented Apr 26, 2017

You should run the process as root. This is not increasing security but merely annoying.

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Apr 26, 2017

@Levino Elasticsearch has checks that will not allow running it as root. This has been discussed in https://discuss.elastic.co/t/why-is-it-elasticsearch-is-not-allowed-to-run-as-root/60413/2 and running processes as root inside a docker container is not a best practice. You can see some reasons in the aforementioned discussion, in https://forums.docker.com/t/root-user-or-non-root-user-inside-container/966/10, http://blog.dscpl.com.au/2015/12/don-run-as-root-inside-of-docker.html and other places.
The risks are especially profound for containers like Elasticsearch where users frequently bind mount host directories with write access.

@vikramvi

This comment has been minimized.

Copy link

commented Oct 26, 2017

@Madhu1512 can you please share your solution in detail, coming from Mac, Windows world; I'm find difficult to understand your answer

@vikramvi

This comment has been minimized.

Copy link

commented Oct 26, 2017

@dliappis I am facing exactly same issue when trying to run https://github.com/Verkurkie/timings-docker/blob/master/docker-compose.yml on Ubuntu 16.4.

I'm new to Ubuntu OS please let me know how to fix this ? Thanks in advance.

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2017

@vikramvi Are you trying to build your own images (given the build context) or just use the images of this repo?

If the later, please start with the docker-compose example in the docs, as you'll need to at least specify the image: section.

@vikramvi

This comment has been minimized.

Copy link

commented Oct 26, 2017

@dliappis I'm trying to use existing images.

As per your comment , please clarify how to achieve it ?

The image runs Elasticsearch using the user elasticsearch (uid: 1000, gid: 1000) so please ensure the directory you are trying to mount is writable by this user.

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2017

@vikramvi First, please, start with the provided example in the docs and make sure that docker-compose up works and that you can access elasticsearch with curl (curl -u elastic:changeme localhost:9200).

After that, you can change the named volume declarations to something like:

    volumes:
      - $PWD/esdir1:/usr/share/elasticsearch/data

...

    volumes:
      - $PWD/esdir2:/usr/share/elasticsearch/data

And create esdir1 and esdir2 in the directory where your docker-compose resides.
The only additional thing you need to do is chgrp 1000 esdir1 esdir2 and chmod g+rwx esdir1 esdir2.

Then docker-compose up should get you a working cluster with bind-mounted directories from your host (you can inspect the data under those new directories).

With this successfully setup, you can try integrate it to your elastic+kibana+other image docker-compose
example.

Also kindly note that the GitHub issues should be primarily used for reporting bugs; for general advice it's best to raise issues in https://discuss.elastic.co

@vikramvi

This comment has been minimized.

Copy link

commented Oct 26, 2017

@dliappis I got this issue resolved with sudo chown -R ubuntuUserName:ubuntuUserName elasticsearch

Please clarify why do I need to do above steps mentioned by you and how above single line solved the issue without any more complex settings ?

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2017

@vikramvi Glad you got it sorted out. I am not sure what is the UID/GID number associated with your ubuntuUserName (should be 1000). If the particular user happened to have a different UID, it wouldn't work, so it's best to not leave things to chance. Also you don't need to alter permissions for the owner, just group permissions are enough. This is achieved with the chgrp + chmod commands above.

The steps re: starting from the docker-compose file in the docs, outlined in my earlier comment, are obviously not mandatory, however they are really helpful to troubleshoot docker issues because they start from a well known, working docker-compose example and show the difference between named volumes and bind mounts. Especially with Docker it is important to read the image docs and understand the setup you are trying to achieve. Unfortunately it also assumes deeper understanding of certain Unix concepts, like the file permission model, especially when using bind-mounts.

@vikramvi

This comment has been minimized.

Copy link

commented Oct 26, 2017

@dliappis Thanks for detailed info. I will read more about it and try to understand more in depth concepts of both docker and unix.

@leosco

This comment has been minimized.

Copy link

commented Feb 7, 2018

I'm still having this issue, even though the directory has the proper permissions.

Please see https://forums.docker.com/t/specify-data-directory-permissions-for-docker-containers-like-elasticsearch/45945

Even with full rwx for everyone, it still doesn't work :/

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2018

@leosco reading https://forums.docker.com/t/specify-data-directory-permissions-for-docker-containers-like-elasticsearch/45945/5 I think you found a solution? Also it's not clear which elasticsearch docker image you are using. It's best that you open a new issue, preferably in discuss.elastic.co, as GH issues are primarily reserved for verified issues and bug reports, with the needed info supplied if you are still having issues.

@jon-torodash

This comment has been minimized.

Copy link

commented Feb 25, 2018

Some of us using PaaS don't have full enough control over the host machines to make the necessary chown invocations to follow the suggestions here, and certainly not named volume support. Do you have any other recommendations?

@leosco

This comment has been minimized.

Copy link

commented Feb 26, 2018

@dimitris-athanasiou I did find a solution. Really, it's apparent to me that docker just has a kinda steep learning curve, but I'm figuring it out.

@jon-torodash basically the docker run -v option will symbolically map a path that the container uses internally to a path you specify. As long as you can do something like -v /home/my/path:/data/db for example, where you have the necessary permissions in your path. Notice the form is real path:path in container, so when you mount a volume, the path in the container becomes a variable that points to the real path you specified. If you're pointing it to a directory you own, even on a PaaS system where you don't have admin privileges, it should work for you as long as you can run docker.

Bottom line: you only need to have permissions in the first param before the colon, the real path i.e /home/mypath/data which is a variable you control. You don't need to have permissions in the /usr/share/elasticsearch path because it's not even going to be used. You're in a way rewriting that path to point to your path instead. I think people are getting confused mostly because the order is reversed. It's real_path:container_path as opposed to the other way around, and if you mount a volume, the container path isn't going to be used at all on the host machine. It's being rerouted to your path on the host machine by docker.

@masudianpour

This comment has been minimized.

Copy link

commented Mar 4, 2018

In addition, in order to solve problem, you can use local-persist plugin.

  1. install local-persist volume plugin (https://github.com/CWSpear/local-persist) (which is also available in official docker docs)
  2. in your ES service volume: - esdata:/usr/share/elasticsearch/data
  3. in volumes section:
volumes:
  esdata:
    driver: local-persist
    driver_opts:
      mountpoint: /your/host/path/to

That's all.

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2018

There shouldn't be a need to use a dedicated storage plugin to have persistence with the elasticsearch-docker image. You can use a docker named volume, as shown in the example in the docs.

The equivalent docker cli command is:

docker run -v esdata:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:6.2.2

Following this you can verify the presence of a dedicated storage volume with docker volume ls (should be called esdata). This is uses the local volume driver and the data is stored locally (by default under /var/lib/docker/volumes/esdata). When using named or anonymous local volumes default permissions should work without issues.

@jon-torodash

This comment has been minimized.

Copy link

commented Mar 6, 2018

Bottom line: you only need to have permissions in the first param before the colon, the real path i.e /home/mypath/data which is a variable you control

Right: lacking those permissions is exactly the problem I've been having, to which is I was hoping some clever alternative existed. I can't "run docker" in this environment except for passing along a compose YAML, with limited expressiveness (e.g. no declaration of network). I can declare volumes, but the host machine is transient. I don't even know under what user the compose command is even being run by the service.

@dliappis

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2018

@jon-torodash for your specific case you can also try the TAKE_FILE_ONWERSHIP env var as described in Note 1

The particular note also describes another possibility by adjusting the gid of the local dir (this is eg possible in k8s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.