Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistent data #27

Closed
FPierre opened this issue May 25, 2015 · 12 comments
Closed

Persistent data #27

FPierre opened this issue May 25, 2015 · 12 comments

Comments

@FPierre
Copy link

FPierre commented May 25, 2015

Hello

I use your awesome Dockerfile Elasticsearch (with Marvel). The only problem is that data are not saved after docker restart (or laptop reboot)

Is there a possibility to save the data to avoid to fill Elasticsearch at every restart ?

I just read the Dockerfile, /data and /conf are not created. There needs to use sudo somewhere?

@itzg
Copy link
Owner

itzg commented May 25, 2015

Thanks for the feedback :)

Unless you're using the --rm option, the container's filesystem should be persistent. As of Docker 1.2 containers don't auto-restart (by default), so do a docker ps -a and see if your container is there, but stopped. If so, you can start the container back with its pre-existing data with docker start ...col #1 from ps...

Another option is to use the new restart policy option. For example, here's how I start one of my nodes in my cluster to always restart when the Docker daemon starts (such as during host reboot):

docker run -d --log-driver=syslog --restart=always -e CLUSTER=nuc \
  -p 9302:9300 \
  --name es-02 -e NODE_NAME=es-02 \
  -e PUBLISH_AS=192.168.0.5:9302 -e UNICAST_HOSTS=192.168.0.5:9300 \
  itzg/elasticsearch

Regarding /conf and /data, the VOLUME directive will instruct Docker to auto-create those (empty) directories if not attached with -v. Here's how my es-02 container looks:

geoff@nuc:/etc/network$ docker exec -it es-02 bash
elasticsearch@5ce03b61896d:~$ ls -l /conf
total 20
-rw-r--r-- 1 elasticsearch elasticsearch 13476 May 16 03:15 elasticsearch.yml
-rw-r--r-- 1 elasticsearch elasticsearch  2030 May 16 03:15 logging.yml
elasticsearch@5ce03b61896d:~$ ls -l /data
total 1248
drwxr-xr-x 3 elasticsearch elasticsearch   4096 May 16 03:15 nuc
-rw-r--r-- 1 elasticsearch elasticsearch 985679 May 25 02:32 nuc.log
-rw-r--r-- 1 elasticsearch elasticsearch   1364 May 16 03:15 nuc.log.2015-05-16
-rw-r--r-- 1 elasticsearch elasticsearch 260783 May 19 23:40 nuc.log.2015-05-19
-rw-r--r-- 1 elasticsearch elasticsearch   2172 May 20 23:34 nuc.log.2015-05-20
-rw-r--r-- 1 elasticsearch elasticsearch   2172 May 21 23:34 nuc.log.2015-05-21
-rw-r--r-- 1 elasticsearch elasticsearch   1916 May 22 23:34 nuc.log.2015-05-22
-rw-r--r-- 1 elasticsearch elasticsearch   2172 May 23 23:34 nuc.log.2015-05-23
-rw-r--r-- 1 elasticsearch elasticsearch   2707 May 24 14:46 nuc.log.2015-05-24
-rw-r--r-- 1 elasticsearch elasticsearch      0 May 16 03:15 nuc_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch      0 May 16 03:15 nuc_index_search_slowlog.log

If you're still having issues, tell me more about your environment. Since you mentioned running on your laptop, is it a boot2docker setup?

@FPierre
Copy link
Author

FPierre commented May 26, 2015

Thanks for those informations.

I think the problem is not here: the directories /data and /conf are missing (even after complete Dockerfile reset). Is it possible that there is a permissions problem?

My / directory:

/$ ls -al
total 116
drwxr-xr-x  24 root root  4096 mai   26 09:38 .
drwxr-xr-x  24 root root  4096 mai   26 09:38 ..
drwxr-xr-x   2 root root 12288 mai   19 08:39 bin
drwxr-xr-x   4 root root  4096 mai   19 08:39 boot
drwxr-xr-x   3 root root  4096 avril 27 14:57 build
drwxrwxr-x   2 root root  4096 avril 27 09:02 cdrom
drwxr-xr-x  18 root root  4300 mai   25 16:46 dev
drwxr-xr-x 145 root root 12288 mai   25 09:09 etc
drwxr-xr-x   4 root root  4096 avril 27 14:40 home
lrwxrwxrwx   1 root root    33 mai    6 09:17 initrd.img -> boot/initrd.img-3.19.0-16-generic
lrwxrwxrwx   1 root root    33 avril 27 09:47 initrd.img.old -> boot/initrd.img-3.19.0-15-generic
drwxr-xr-x  24 root root  4096 avril 27 09:45 lib
drwxr-xr-x   2 root root  4096 avril 27 09:42 lib64
drwx------   2 root root 16384 avril 27 09:00 lost+found
drwxr-xr-x   3 root root  4096 avril 27 09:07 media
drwxr-xr-x   2 root root  4096 oct.  16  2014 mnt
drwxr-xr-x   5 root root  4096 avril 27 10:58 opt
dr-xr-xr-x 255 root root     0 mai   25 09:43 proc
drwx------   5 root root  4096 mai   25 10:35 root
drwxr-xr-x  27 root root   820 mai   26 09:02 run
drwxr-xr-x   2 root root 12288 mai   19 08:39 sbin
drwxr-xr-x   2 root root  4096 oct.  22  2014 srv
dr-xr-xr-x  13 root root     0 mai   25 12:02 sys
drwxrwxrwt  13 root root  4096 mai   26 09:37 tmp
drwxr-xr-x  10 root root  4096 oct.  22  2014 usr
drwxr-xr-x  14 root root  4096 avril 27 10:38 var
lrwxrwxrwx   1 root root    30 mai    6 09:17 vmlinuz -> boot/vmlinuz-3.19.0-16-generic
lrwxrwxrwx   1 root root    30 avril 27 09:47 vmlinuz.old -> boot/vmlinuz-3.19.0-15-generic

I use the Dockerfile on my Ubuntu laptop without boot2docker (official Docker installation)

@itzg
Copy link
Owner

itzg commented May 26, 2015

Excellent, Ubuntu is my primary host platform for using Docker. Can you go through the following sequence, reply with the output from each step and this will help me narrow down the symptoms:

docker pull itzg/elasticsearch
docker run -d -p 9292:9200 --name es-92 itzg/elasticsearch
curl http://localhost:9292
docker exec es-92 ls -l /data
docker restart es-92
docker exec es-92 ls -l /data
docker inspect -f "{{.Volumes}}" es-92

@itzg
Copy link
Owner

itzg commented May 29, 2015

@FPierre , have you been able to try out the latest test case?

@FPierre
Copy link
Author

FPierre commented Jun 3, 2015

Sorry, I'm not able to access to my main computer for another 5 days. I could answer Tuesday

@FPierre
Copy link
Author

FPierre commented Jun 22, 2015

@itzg, sorry I was an another project.
I try out your test case, and it's work! Remove and install the image solved the problem.

Good to me :)

@FPierre FPierre closed this as completed Jun 22, 2015
@itzg
Copy link
Owner

itzg commented Jun 22, 2015

Glad to hear and thanks for following up.

@rdpanek
Copy link

rdpanek commented Aug 22, 2015

Hello @itzg , thanks for your image ;-) So, I have issue with persist /data and /config: My goal is - share directory /data and /config between host (ubuntu) and container. When stop or restart container, data and config must be persist on host computer and when start new container with --volumes option, must be mounted to new running container.

docker run -d --name elastic -p 9200:9200 -p 9300:9300 -v /absolute/host/directory:/data itzg/elasticsearch
9830c3e7c9f7eb5d3c853a35a6b071e92dd7fb54b509c20d2bab1e4303643111

and get logs

$ docker logs elastic
Starting Elasticsearch with the options -Des.path.conf=/conf   -Des.path.data=/data   -Des.path.logs=/data   -Des.transport.tcp.port=9300   -Des.http.port=9200
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /data/elasticsearch.log (Permission denied)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:112)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:131)
    at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:216)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
log4j:ERROR Either File or DatePattern options are not set for appender [file].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /data/elasticsearch_index_indexing_slowlog.log (Permission denied)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:112)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:131)
    at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:216)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
log4j:ERROR Either File or DatePattern options are not set for appender [index_indexing_slow_log_file].
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /data/elasticsearch_index_search_slowlog.log (Permission denied)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:112)
    at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:131)
    at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:216)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
log4j:ERROR Either File or DatePattern options are not set for appender [index_search_slow_log_file].
[2015-08-22 19:42:11,773][INFO ][node                     ] [Speedball] version[1.7.0], pid[8], build[929b973/2015-07-16T14:31:07Z]
[2015-08-22 19:42:11,775][INFO ][node                     ] [Speedball] initializing ...
[2015-08-22 19:42:11,962][INFO ][plugins                  ] [Speedball] loaded [], sites []
{1.7.0}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to created node environment]
    AccessDeniedException[/data/elasticsearch]

Have you some idea, how to solutions? Thanks

@rdpanek
Copy link

rdpanek commented Aug 22, 2015

docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:35:49 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:35:49 UTC 2015
 OS/Arch:      linux/amd64

@itzg
Copy link
Owner

itzg commented Aug 22, 2015

In order to run ES as a non-root user, the container useradd's "elasticsearch" (with uid=1000). I believe the mapped host directory also needs to be write-able by uid=1000. On Ubuntu that works out since my main login also got uid=1000 :).

With that said, I'll do some research, because this has always felt awkward to me.

@itzg
Copy link
Owner

itzg commented Aug 22, 2015

...already found there's healthy demand and discussion about this moby/moby#7198

@rdpanek
Copy link

rdpanek commented Aug 22, 2015

@itzg Thanks, now it's ok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants