Logrotate configuration and /etc/init.d/gitlab restart issue. #52

Closed
TiagoTT opened this Issue Jan 24, 2013 · 1 comment

Projects

None yet

1 participant

@TiagoTT
TiagoTT commented Jan 24, 2013

Hi!

I have done this for my installation and would like to share and ask for advise.
The file is /etc/logrotate.d/gitlab and has the following content:

/home/gitlab/gitlab/log/application.log
/home/gitlab/gitlab/log/development.log
/home/gitlab/gitlab/log/githost.log
/home/gitlab/gitlab/log/production.log
/home/gitlab/gitlab/log/sidekiq.log
/home/gitlab/gitlab/log/unicorn.stderr.log
/home/gitlab/gitlab/log/unicorn.stdout.log
{
    rotate 7
    daily
    maxsize 10M
    minsize 1M
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        invoke-rc.d gitlab restart > /dev/null
    endscript
}

The tricky part is the restart, because the reload wasn't enough to make the running processes close and reopen the log files. This works well and the logs rotate as intended.

Logrotate runs daily by crontab. And I noticed that the message queues were left in unusable state after restart, so the email notifications stopped.

I found that stopping and starting sidekiq was enough to get it working again and then I went looking for the cause in the init.d script. It seems that running stop and start operations for sidekiq as background processes caused some kind of race condition.

So to fix this I removed the trailing '&' from the commands in lines 77 and 79:

...
sudo -u gitlab -H bash -l -c "mkdir -p $PID_PATH && $STOP_SIDEKIQ  > /dev/null  2>&1 &"
....
sudo -u gitlab -H bash -l -c "mkdir -p $PID_PATH && $START_SIDEKIQ  > /dev/null  2>&1 &"
...

Could anyone check if this problem can safely be fixed this way?

@TiagoTT
TiagoTT commented Feb 19, 2013

I will drop this issue and start using a different configuration, based on the following one, which does not require the application restart (see option copytruncate in logrotate manpage).

https://github.com/cvut/gentoo-overlay/blob/master/www-apps/gitlabhq/files/gitlab.logrotate

@TiagoTT TiagoTT closed this Feb 19, 2013
@hiroponz hiroponz added a commit to hiroponz/gitlab-recipes that referenced this issue Apr 8, 2013
@hiroponz hiroponz Fix not starting sidekiq process if "gitlab restart".
It needs to wait for completing STOP_SIDEKIQ before START_SIDEKIQ.
refs #52
7f8dd74
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment