[Bug] PID_FILE not set and not created #9133

gerhard-tinned · 2017-08-31T09:00:07Z

What Grafana version are you using?

Installed Packages
Name        : grafana
Arch        : x86_64
Version     : 4.4.3
Release     : 1
 
via yum repository at baseurl https://packagecloud.io/grafana/stable/el/7/$basearch

What OS are you running grafana on?

CentOS Linux release 7.3.1611 (Core)

What did you do?
I tried to setup monitoring for the grafana-server. To do this I was looking for the pid file.
What was the expected result?
I expected to find a pid file in the configured path or at "/var/run/...".
What happened instead?
There was no pid file found on the system.
Detailed Description

Grafana was installed using yum/rpm from the grafana repository. The system is started/stopped via systemd (systemctl command). According to the systemd unit file, the pidfile is set from a variable which is loaded from "/etc/sysconfig/grafana-server".

[Service]
EnvironmentFile=/etc/sysconfig/grafana-server
User=grafana
Group=grafana
Type=simple
Restart=on-failure
WorkingDirectory=/usr/share/grafana
ExecStart=/usr/sbin/grafana-server                                \
                            --config=${CONF_FILE}                 \
                            --pidfile=${PID_FILE}                 \
                            cfg:default.paths.logs=${LOG_DIR}     \
                            cfg:default.paths.data=${DATA_DIR}    \
                            cfg:default.paths.plugins=${PLUGINS_DIR}

The referenced file does not even contain the variable "PID_FILE" nor does it set a value.

A search through the documentation for the grafana.ini did not reveal any possibility to configure the pidfile.

Solution & Workaround

The workaround is the same as the solution. The config file /etc/sysconfig/grafana-server should include the PID_FILE variable and set it to a default location like this.

PID_FILE=/var/run/grafana.pid

This setting would have been expected to be in the file deployed via the rpm.

The text was updated successfully, but these errors were encountered:

gerhard-tinned · 2017-08-31T09:42:21Z

It seems when this option is set, grafana server does not start. Interestingly without any error message in the grafana.log

daniellee · 2017-08-31T09:54:31Z

As you said, looks like the PID_FILE variable is not set here:

https://github.com/grafana/grafana/blob/master/packaging/rpm/sysconfig/grafana-server

Does it work if you use this instead:

PID_FILE=/var/run/grafana-server.pid

Anyway, looks like a bug.

gerhard-tinned · 2017-08-31T11:20:11Z

No, it does not work. Even if i touch the pid file before I start.

Interesting at this point is the way the arguments are passed to the grafana-server.

The systemd unit file lists this:

ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE} cfg:default.paths.logs=${LOG_DIR} cfg:default.paths.data=${DATA_DIR} cfg:default.paths.plugins=${PLUGINS_DIR}

The help output shows this:

$ /usr/sbin/grafana-server --help
Usage of /usr/sbin/grafana-server:
  -config string
        path to config file
  -homepath string
        path to grafana install/home path, defaults to working directory
  -pidfile string
        path to pid file
  -profile
        Turn on pprof profiling
  -profile-port int
        Define custom port for profiling (default 6060)
  -v    prints current version and exits

(notice the help screen "-pidfile" while the unit file passes "--pidfile". I tried adopting the unitfile for this but the behaviour did not change at all)

gerhard-tinned · 2017-08-31T17:30:45Z

I was digging deeper. The problem is actually quite simple. But as the whole Grafana is deployed via rpm, the related installation procedure in the pre- and post-script section needs to be extended by just a a few lines. (I just don't know where the rpm spec file can be found)

What is the problem?
1.) The grafana-server is just "not starting" as many would report it. Usually you would expect some more details but the grafana-server dies without a single character of log or output! - This is definitive a BUG and should be fixed!

2.) The root cause (really could be easier found with some error messages) is a plain simple permission problem caused by the strange behavior of starting the daemon.

The init-script used in non-systemd versions, creates the pid file before the daemon is started with the privileges of the init-script (root usually) and changes its ownership to the user/group configured to start the daemon.

grafana/packaging/rpm/init.d/grafana-server

Line 86 in 40ae2ce

touch "$PID_FILE" && chown "$GRAFANA_USER":"$GRAFANA_GROUP" "$PID_FILE"

Now with CentOS/RHEL 7 systemd is starting the grafana-server with reduced priviledges. This causes the gravana-server to fail while trying to create the pid file under (for example /var/run/). ==> Again, this forensic search for the root cause would have been way faster with a error message from grafana-server.

grafana/packaging/rpm/systemd/grafana-server.service

Lines 10 to 11 in 40ae2ce

    
           User=grafana 
        
           Group=grafana

How to fix it?

The rpm should create a directory under /var/run/ like "/var/run/grafana/". This directories owner and group can be changed to the daemons user and group.
The rpm should provide the "PID_FILE" setting in the /etc/sysconfig/grafana-serve with content like this. PID_FILE=/var/run/grafana/grafana-server.pid

If this procedure is applied manually, the grafana-server starts properly with a pid file as the directory the pid file should be created in is owned by the daemons user which allows the creation of files.

I am not familiar with the packaging process used here but I guess the related lines should be added around this line ...

grafana/packaging/rpm/control/postinst

Line 47 in 40ae2ce

# Set user permissions on /var/log/grafana, /var/lib/grafana

To activate this only on systemd machines, a simple check like it is done a little further down would be enough.

if [ -x /bin/systemctl ] ; then
  mkdir -p /var/run/grafana
  chown -R $GRAFANA_USER:$GRAFANA_GROUP /var/run/grafana
fi

I assume the same needs to be done for the deb packages as they use systemd as well in resent releases.

If my assumptions about the fix are correct I would be happy to submit a pull request if you want.

torkelo · 2017-08-31T19:20:29Z

Thanks for digging deeper into this. The issue with no log output when shutting down is strange, have never seen that not seen it reported, always reported via grafana log when shutting down

gerhard-tinned · 2017-09-01T06:14:15Z

Well, when grafana-server is shut down, there are a few log messages usually.

It seems as if it is just in this special case where he cannot create the pidfile. If I would need to guess, I would think grafana-server is trying to create the pid before any log or output was generated and dies without an error massage at that point.

It's less of an issue that the logfile does not show any error. The biggest issue is the missing output of the dieing process (stdout/stderr). If there would have been one, systemd would have wrote it to the journal. Or you would see it while trying to start the daemon manually.

torkelo · 2017-09-01T08:52:17Z

oh, that is probably it, it dies before any logging is setup. There are some logging calls if pid directory does not exist or pid file creation fails but because the logging is not configured at that point it probably goes nowhere

gerhard-tinned · 2017-09-04T07:05:12Z

Before it goes nowhere, stderr would be a nice place to dump those messages. :)

torkelo · 2017-09-04T08:45:20Z

I hope I fixed the pid issue (and improved the logging) in c3cffeb

introduce a PID_FILE_DIR env variable that is used on systemd systems (post install script creates the dir and sets the permission, the systemd service spec uses it to set --pidfile argument). This way there is no change for non systemd systems.

gerhard-tinned · 2017-09-04T13:25:26Z

Hi, I am really sorry but I was digging deeper into the complete issue and realized something else. On CentOS7, per default the /var/run/ is a link to /run which is a tempfs and therefore all files and directories will not survive a reboot.

That means the changes in the "postinst" scripts will not have any effect until the server was rebooted. The changes to create the /var/run/grafana/ directory needs to be added to the systemd unit file.

As the grafana-server is already started by systemd with droped priiviledges, the directory needs to be created before the start from systemd. I found the following possibility.

This will instruct systemd to start the daemon with the user and group.

grafana/packaging/deb/systemd/grafana-server.service

Lines 8 to 20 in 4eabf6c

    
           [Service] 
        
           EnvironmentFile=/etc/default/grafana-server 
        
           User=grafana 
        
           Group=grafana 
        
           Type=simple 
        
           Restart=on-failure 
        
           WorkingDirectory=/usr/share/grafana 
        
           ExecStart=/usr/sbin/grafana-server                                        \ 
        
                                       --config=${CONF_FILE}                         \ 
        
                                       --pidfile=${PID_FILE_DIR}/grafana-server.pid  \ 
        
                                       cfg:default.paths.logs=${LOG_DIR}             \ 
        
                                       cfg:default.paths.data=${DATA_DIR}            \ 
        
                                       cfg:default.paths.plugins=${PLUGINS_DIR}

When changed according to this example, systemd will execute the two commands (as root) before starting the daemon. That should ensure the rights to create the var/run/grafana/ directory before the daemon is started.

 [Service] 
 EnvironmentFile=/etc/default/grafana-server 
 User=grafana 
 Group=grafana 
 Type=simple 
 Restart=on-failure 
 WorkingDirectory=/usr/share/grafana 
 PermissionsStartOnly=true
 ExecStartPre=-/usr/bin/mkdir ${PID_FILE_DIR}
 ExecStartPre=/usr/bin/chown -R grafana:grafana ${PID_FILE_DIR}
 ExecStart=/usr/sbin/grafana-server                                        \ 
                             --config=${CONF_FILE}                         \ 
                             --pidfile=${PID_FILE_DIR}/grafana-server.pid  \ 
                             cfg:default.paths.logs=${LOG_DIR}             \ 
                             cfg:default.paths.data=${DATA_DIR}            \ 
cfg:default.paths.plugins=${PLUGINS_DIR}

Together with the new introduced variable for the PID directory, that should work.

torkelo · 2017-09-04T14:30:54Z

Maybe we can use

RuntimeDirectory=grafana
RuntimeDirectoryMode=0750

?

…9133

torkelo · 2017-09-04T14:41:44Z

Seems to work on ubuntu 16 atleast, and is the approach elasticsearch is using
https://github.com/elastic/elasticsearch/blob/master/distribution/src/main/packaging/systemd/elasticsearch.service#L8

gerhard-tinned · 2017-09-04T17:17:06Z

And where should this runtime directory be?? It reads like a relative path. But from where??

gerhard-tinned · 2017-09-04T17:23:39Z

https://blog.hqcodeshop.fi/archives/93-Handling-varrun-with-systemd.html

There is even the possibility to specify the PIDFile variable for systemd. Depending on the behaviour of the daemon, this might make sense. See details here: https://lists.debian.org/debian-user/2016/10/msg00422.html

gerhard-tinned · 2017-09-05T07:49:04Z

Thanks @torkelo, I did some reading on what you suggested. It seems this setting is exactly the option ment to do what we need. See detailed description of the setting here. At first I did not know where the directory would be created. But it seems the name is relative to the /run/ directory.

https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=

So a directive like the following would do the trick.

RuntimeDirectory=grafana

Thanks a lot @torkelo

imports a fix that sets a PID_FILE_DIR variable in /etc/defaults/grafana-server see grafana/grafana#9133 see grafana/grafana@c3cffeb#diff-ad598638c92a28d5d7ffdaba305d92b2

gerhard-tinned changed the title ~~Bug~~ [Bug] BID_FILE not set and not created Aug 31, 2017

gerhard-tinned changed the title ~~[Bug] BID_FILE not set and not created~~ [Bug] PID_FILE not set and not created Aug 31, 2017

daniellee added type/bug type/build-packaging labels Aug 31, 2017

daniellee added this to the 4.4.4 milestone Aug 31, 2017

torkelo modified the milestones: 4.4.4, 4.5.0 Sep 1, 2017

torkelo closed this as completed in c3cffeb Sep 4, 2017

torkelo added a commit that referenced this issue Sep 4, 2017

packaging: changed how pid directory is created for systemd systems, #…

5d3c3a6

…9133

despairblue mentioned this issue Dec 10, 2017

fix new grafana versions not starting ansiblebit/grafana#7

Closed

kylebrandt mentioned this issue Jun 10, 2019

Pidfile is not removed on exit #17471

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] PID_FILE not set and not created #9133

[Bug] PID_FILE not set and not created #9133

gerhard-tinned commented Aug 31, 2017

gerhard-tinned commented Aug 31, 2017

daniellee commented Aug 31, 2017

gerhard-tinned commented Aug 31, 2017

gerhard-tinned commented Aug 31, 2017 •

edited

torkelo commented Aug 31, 2017

gerhard-tinned commented Sep 1, 2017

torkelo commented Sep 1, 2017

gerhard-tinned commented Sep 4, 2017 •

edited

torkelo commented Sep 4, 2017

gerhard-tinned commented Sep 4, 2017

torkelo commented Sep 4, 2017

torkelo commented Sep 4, 2017

gerhard-tinned commented Sep 4, 2017 •

edited

gerhard-tinned commented Sep 4, 2017

gerhard-tinned commented Sep 5, 2017

[Bug] PID_FILE not set and not created #9133

[Bug] PID_FILE not set and not created #9133

Comments

gerhard-tinned commented Aug 31, 2017

gerhard-tinned commented Aug 31, 2017

daniellee commented Aug 31, 2017

gerhard-tinned commented Aug 31, 2017

gerhard-tinned commented Aug 31, 2017 • edited

torkelo commented Aug 31, 2017

gerhard-tinned commented Sep 1, 2017

torkelo commented Sep 1, 2017

gerhard-tinned commented Sep 4, 2017 • edited

torkelo commented Sep 4, 2017

gerhard-tinned commented Sep 4, 2017

torkelo commented Sep 4, 2017

torkelo commented Sep 4, 2017

gerhard-tinned commented Sep 4, 2017 • edited

gerhard-tinned commented Sep 4, 2017

gerhard-tinned commented Sep 5, 2017

gerhard-tinned commented Aug 31, 2017 •

edited

gerhard-tinned commented Sep 4, 2017 •

edited

gerhard-tinned commented Sep 4, 2017 •

edited