cannot start influxdb #4824

airyland · Nov 17, 2015

The service just stop two days ago and I couldn't start or restart it. The status always returned FAIL. After I upgrade to 0.9.5, it still failed but was with some logs:

sudo service influxdb restart

2015/11/18 03:05:36 InfluxDB starting, version 0.9.5-rc3, branch 0.9.5, commit a589f941e93078a4a093806d7336ee903d32f23a, built 2015-11-17T00:42:10+0000
2015/11/18 03:05:36 Go version go1.4.2, GOMAXPROCS set to 2
2015/11/18 03:05:36 Using configuration at: /etc/influxdb/influxdb.conf
[metastore] 2015/11/18 03:05:36 Using data dir: /var/opt/influxdb/meta
[metastore] 2015/11/18 03:05:36 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[localhost:8088]
[metastore] 2015/11/18 03:05:36 Node at localhost:8088 [Follower]
[metastore] 2015/11/18 03:05:37 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2015/11/18 03:05:37 spun up monitoring for 1
[store] 2015/11/18 03:05:37 Using data dir: /var/opt/influxdb/data
[metastore] 2015/11/18 03:05:37 Updated node id=1 hostname=localhost:8088
[registration] 2015/11/18 03:05:39 registration service terminating
[retention] 2015/11/18 03:05:39 retention policy enforcement terminating
[monitor] 2015/11/18 03:05:39 shutting down monitor system
[handoff] 2015/11/18 03:05:39 shutting down hh service
[subscriber] 2015/11/18 03:05:39 closed service
run: open server: open tsdb store: failed to open shard 102: load metadata index: open /var/opt/influxdb/wal/bz_shop/default/102/000000.meta: permission denied

And when I ran influxd

influxd run --config /etc/influxdb/influxdb.conf

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

2015/11/18 03:14:01 InfluxDB starting, version 0.9.5-rc3, branch 0.9.5, commit a589f941e93078a4a093806d7336ee903d32f23a, built 2015-11-17T00:42:10+0000
2015/11/18 03:14:01 Go version go1.4.2, GOMAXPROCS set to 2
2015/11/18 03:14:01 Using configuration at: /etc/influxdb/influxdb.conf
[metastore] 2015/11/18 03:14:01 Using data dir: /var/opt/influxdb/meta
[metastore] 2015/11/18 03:14:01 Node at localhost:8088 [Follower]
[metastore] 2015/11/18 03:14:01 Skipping cluster join: already member of cluster: nodeId=1 raftEnabled=true peers=[localhost:8088]
[metastore] 2015/11/18 03:14:02 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2015/11/18 03:14:02 spun up monitoring for 1
[store] 2015/11/18 03:14:02 Using data dir: /var/opt/influxdb/data
[metastore] 2015/11/18 03:14:02 Updated node id=1 hostname=localhost:8088
[wal] 2015/11/18 03:14:04 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2015/11/18 03:14:04 WAL writing to /var/opt/influxdb/wal/bz_shop/default/102
[wal] 2015/11/18 03:14:05 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2015/11/18 03:14:05 WAL writing to /var/opt/influxdb/wal/bz_shop/default/1539
[wal] 2015/11/18 03:14:06 WAL starting with 30720 ready series size, 0.50 compaction threshold, and 52428800 partition size threshold
[wal] 2015/11/18 03:14:06 WAL writing to /var/opt/influxdb/wal/bz_shop/default/2473
.
.
.
Killed

otoolep · Nov 17, 2015

@benbjohnson -- any idea why this would happen? Apart from some simple permission problem?

@airyland -- can you show us your directory and file tree at that point? With full permission details etc?

benbjohnson · Nov 17, 2015

@otoolep It looks like /var/opt/influxdb/wal/bz_shop/default/102/000000.meta doesn't have read & write permission for the user running influxd.

otoolep · Nov 17, 2015

Yep, it looks as simple as that.

@airyland -- any chance you ran influxd as a different user at some point?

airyland · Nov 18, 2015

@otoolep Actually I only use user root.. Before the error came out there was no any log for the failure of starting.

airyland · Nov 18, 2015

I use "chmod" to fix the permission and still failed. So why did it just terminate the service without any reason?

[metastore] 2015/11/18 11:06:15 Node at localhost:8088 [Follower]
[metastore] 2015/11/18 11:06:17 Node at localhost:8088 [Leader]. peers=[localhost:8088]
[metastore] 2015/11/18 11:06:17 spun up monitoring for 1
[store] 2015/11/18 11:06:17 Using data dir: /var/opt/influxdb/data
[metastore] 2015/11/18 11:06:17 Updated node id=1 hostname=localhost:8088
[registration] 2015/11/18 11:06:17 registration service terminating
[retention] 2015/11/18 11:06:17 retention policy enforcement terminating
[monitor] 2015/11/18 11:06:17 shutting down monitor system
[handoff] 2015/11/18 11:06:17 shutting down hh service
[subscriber] 2015/11/18 11:06:17 closed service

airyland · Nov 18, 2015

I upgraded to 0.9.6 and ran "influxd run ..." and it worked...
But service influxdb restart still failed.

So I don't know what happened..

yuokada · Nov 19, 2015

this issue happened in my enviroment.
My env is CnetOS6.7, InfluxDB v0.9.5-rc2

$ rpm -q  influxdb
influxdb-0.9.5-0.1.rc2.x86_64

I run influxdb by root user only, too.

rossmcdonald · Nov 19, 2015

@airyland I believe your permissions are incorrect due running InfluxDB as root. Can you send us the output from the following command?

sudo find /var/opt/influxdb -user root

It is also highly recommended that you not run InfluxDB as the root user.

yuokada · Nov 20, 2015

In passing, my results is here.

$ sudo find /var/lib/influxdb -user root
/var/lib/influxdb/wal/_internal/monitor/10/000000.meta
/var/lib/influxdb/wal/_internal/monitor/1/000000.meta
/var/lib/influxdb/wal/stress/default/11/000000.meta
/var/lib/influxdb/wal/mydb/default/6/000000.meta
/var/lib/influxdb/wal/mydb/default/7/000000.meta
/var/lib/influxdb/wal/mydb/default/9/000000.meta
/var/lib/influxdb/wal/mydb/default/5/000000.meta
/var/lib/influxdb/wal/mydb/default/8/000000.meta

I change these files owner to influxdb and server running fine.

airyland · Nov 20, 2015

@rossmcdonald

root@iseekr:~# sudo find /var/opt/influxdb -user root
/var/opt/influxdb/wal/bz_bbs/default/49/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/33/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/9/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/23/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/7/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/25/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/514/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/18/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/1535/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/2484/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/32/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/20/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/26/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/16/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/24/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/8/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/1/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/2470/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/35/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/5/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/13/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/6/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/29/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/3/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/22/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/34/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/4/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/36/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/12/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/19/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/28/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/15/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/2/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/31/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/17/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/30/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/27/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/14/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/11/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/21/000000.meta
/var/opt/influxdb/wal/bz_bbs/default/10/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2551/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2542/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/519/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2529/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2577/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/1540/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/96/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2474/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/100/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2488/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2497/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/2565/000000.meta
/var/opt/influxdb/wal/bz_doctor/default/94/000000.meta
/var/opt/influxdb/wal/mome/default/2505/000000.meta
/var/opt/influxdb/wal/mome/default/2503/000000.meta
/var/opt/influxdb/wal/mome/default/2504/000000.meta
/var/opt/influxdb/wal/mome/default/2511/000000.meta
/var/opt/influxdb/wal/mome/default/2506/000000.meta
/var/opt/influxdb/wal/mome/default/2533/000000.meta
/var/opt/influxdb/wal/mome/default/2510/000000.meta
/var/opt/influxdb/wal/mome/default/2507/000000.meta
/var/opt/influxdb/wal/mome/default/2508/000000.meta
/var/opt/influxdb/wal/mome/default/2509/000000.meta
/var/opt/influxdb/wal/mome/default/2544/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2576/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2528/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2550/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2496/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2564/000000.meta
/var/opt/influxdb/wal/bz_shop/default/102/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2473/000000.meta
/var/opt/influxdb/wal/bz_shop/default/1539/000000.meta
/var/opt/influxdb/wal/bz_shop/default/99/000000.meta
/var/opt/influxdb/wal/bz_shop/default/518/000000.meta
/var/opt/influxdb/wal/bz_shop/default/97/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2487/000000.meta
/var/opt/influxdb/wal/bz_shop/default/2541/000000.meta
/var/opt/influxdb/wal/test/default/2555/000000.meta
/var/opt/influxdb/wal/test/default/39/000000.meta
/var/opt/influxdb/wal/test/default/52/000000.meta
/var/opt/influxdb/wal/test/default/2574
/var/opt/influxdb/wal/test/default/2574/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2578/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2571/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2573/000000.meta
/var/opt/influxdb/wal/_internal/monitor/2572/000000.meta
/var/opt/influxdb/wal/bz_umeng/default/528/000000.meta
/var/opt/influxdb/wal/bz_umeng/default/529/000000.meta
/var/opt/influxdb/wal/mome_activity/default/2483/000000.meta
/var/opt/influxdb/wal/mome_activity/default/2482/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/59/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/62/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/63/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/65/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/516/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2543/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/64/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/60/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/61/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2495/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2486/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2563/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/1537/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/58/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2472/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2527/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2552/000000.meta
/var/opt/influxdb/wal/bz_fkzr/default/2575/000001.meta
/var/opt/influxdb/wal/mome1/default/2514/000000.meta
/var/opt/influxdb/wal/mome1/default/2513/000000.meta
/var/opt/influxdb/wal/mome1/default/2515/000000.meta
/var/opt/influxdb/wal/mome1/default/2517/000000.meta
/var/opt/influxdb/wal/mome1/default/2519/000000.meta
/var/opt/influxdb/wal/mome1/default/2520/000000.meta
/var/opt/influxdb/wal/mome1/default/2516/000000.meta
/var/opt/influxdb/wal/mome1/default/2518/000000.meta
/var/opt/influxdb/wal/mome1/default/2512/000000.meta
/var/opt/influxdb/wal/game_pintu/default/2559/000000.meta
/var/opt/influxdb/wal/game_pintu/default/2556/000000.meta
/var/opt/influxdb/wal/game_pintu/default/2567/000000.meta
/var/opt/influxdb/data/test/default/2574

rossmcdonald · Nov 20, 2015

@airyland Your permissions are incorrect, as those files need to be owned by the influxdb user if you would like to run InfluxDB as a service. To fix, you should just be able to run:

chown -R influxdb:influxdb /var/opt/influxdb

airyland · Nov 20, 2015

@rossmcdonald I see. Thanks!

beckettsean added the area/packaging label Nov 19, 2015

beckettsean assigned rossmcdonald Nov 19, 2015

rossmcdonald removed the area/packaging label Nov 20, 2015

airyland closed this Nov 20, 2015

This was referenced Mar 10, 2016

Closed

Debian systemd error bdangit/chef-influxdb#103

Closed

Permission denied while opening .pem file readable by others #7292

influxdata/influxdb

cannot start influxdb #4824

airyland commented Nov 17, 2015

otoolep commented Nov 17, 2015

benbjohnson commented Nov 17, 2015

otoolep commented Nov 17, 2015

airyland commented Nov 18, 2015

airyland commented Nov 18, 2015

airyland commented Nov 18, 2015

yuokada commented Nov 19, 2015

beckettsean added the area/packaging label Nov 19, 2015

beckettsean assigned rossmcdonald Nov 19, 2015

rossmcdonald commented Nov 19, 2015

yuokada commented Nov 20, 2015

airyland commented Nov 20, 2015

rossmcdonald commented Nov 20, 2015

rossmcdonald removed the area/packaging label Nov 20, 2015

airyland commented Nov 20, 2015

airyland closed this Nov 20, 2015

This was referenced Mar 10, 2016

Debian systemd error bdangit/chef-influxdb#103

Permission denied while opening .pem file readable by others #7292

influxdata/influxdb

Join GitHub today

cannot start influxdb #4824

Comments

airyland commented Nov 17, 2015

otoolep commented Nov 17, 2015

benbjohnson commented Nov 17, 2015

otoolep commented Nov 17, 2015

airyland commented Nov 18, 2015

airyland commented Nov 18, 2015

airyland commented Nov 18, 2015

yuokada commented Nov 19, 2015

beckettsean added the area/packaging label Nov 19, 2015

beckettsean assigned rossmcdonald Nov 19, 2015

rossmcdonald commented Nov 19, 2015

yuokada commented Nov 20, 2015

airyland commented Nov 20, 2015

rossmcdonald commented Nov 20, 2015

rossmcdonald removed the area/packaging label Nov 20, 2015

airyland commented Nov 20, 2015

airyland closed this Nov 20, 2015

This was referenced Mar 10, 2016

Debian systemd error bdangit/chef-influxdb#103

Permission denied while opening .pem file readable by others #7292