docker: rm container + run new one - fails to start if gitlab-ci is enabled #9611

Closed
piec opened this Issue Sep 8, 2015 · 5 comments

Projects

None yet

6 participants

@piec
piec commented Sep 8, 2015

Hi,

Thanks for this great projet.
I recently started using gitlab, in the official docker container. I have just a few projets on it.
I enabled gitlab-ci using ci_external_url, the problem is if I docker rm gitlab and docker run it again then it won't start again correctly. This is a problem if I want to change the run parameters like --restart, volumes or port redirections.
In that case redisabling gitlab-ci doesn't solve the problem. I have to remove the container, disable gitlab-ci and docker run ... again.

I run my container like this:

docker run \               
    --publish 8443:443 --publish 8080:80 --publish 2222:22 \
    --name gitlab \
    --restart always \
    --add-host='gitlab.xxx.com:127.0.0.1' \
    --add-host='ci.xxx.com:127.0.0.1' \
    --volume /srv/gitlab/config:/etc/gitlab \
    --volume /srv/gitlab/logs:/var/log/gitlab \
    --volume /srv/gitlab/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

It should be able to survive this (but doesn't):

docker rm gitlab
docker run [the same as before]

The run log can be found here.
https://gist.github.com/piec/f8d886b6dc3682a45717#file-gitlab-log-L561

Issues seem related to this, but I don't know gitlab enough to understand what happens

2015-09-08_09:53:22.12901           `-._        _.-'                                           
2015-09-08_09:53:22.12902               `-.__.-'                                               
2015-09-08_09:53:22.12902 
2015-09-08_09:53:22.12903 [847] 08 Sep 09:53:22.128 # Server started, Redis version 2.8.21
2015-09-08_09:53:22.12903 [847] 08 Sep 09:53:22.128 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2015-09-08_09:53:22.12903 [847] 08 Sep 09:53:22.128 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2015-09-08_09:53:22.12904 [847] 08 Sep 09:53:22.128 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2015-09-08_09:53:22.12904 [847] 08 Sep 09:53:22.129 # Fatal error loading the DB: Permission denied. Exiting.

Cheers,
Pierre

@Julio-Guerra
Contributor

Same problem, same log since I updated gitlab image today. It seems to be related to an overall access right problem. Most of the access right errors seem to be related to postgresql access rights (can't access the socket, can't access PG_VERSION, etc.). Have you changed user ids? Recreating a container from scratch works.

@tiesselune

Hi! I ran into a similar problem with permissions, regardless of gitlab-ci, when upgrading from 7 to 8.0.4. (by removing the container and re-running it with old volumes after upgrading the image)

At first I did the following:

docker exec gitlab update-permissions
docker restart gitlab

as specified at the end of the documentation on docker official images.

Then I noticed I still had a problem with redis permissions, and noticed dump.rdb still had wrong permissions.

So I ran

docker exec gitlab chown gitlab-redis /var/opt/gitlab/redis/*
docker restart gitlab

and then it worked like a charm.

@sebglon
sebglon commented Nov 12, 2015

Same erro for me and docker exec gitlab chown gitlab-redis /var/opt/gitlab/redis/*

solve problem

but update-permissions not change permission on /var/opt/gitlab/redis/

@bhechinger

So I'm currently in a similar situation (with 8.2.2).

I get the redis permission error. I do:

docker exec gitlab update-permissions
docker exec gitlab chown gitlab-redis /var/opt/gitlab/redis/
docker restart gitlab

But that results in things exploding horribly. I've even tried removing and re-creating the container to no avail.

gitlab   | Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'

I'm so close I can taste it!

The really frustrating part of all of this is the fact that the first time I stood up the container it worked.

I dropped my config into place and it's all been downhill ever since.

Even if I wipe out the .........

And then it just hit me. I stopped the container. I deleted the container. I then proceeded to remove everything that was in the mapped volume for /var/opt/gitlab and now things are all groovy!

@connorshea
Member

Hi! We’re closing our issue tracker on GitHub so we can focus on the GitLab.com issue tracker and respond to issues faster.

If you think this is still an issue I encourage you to open it in our issue tracker on GitLab.com. You can log in on GitLab.com using your GitHub account if you'd like to contribute an issue.

@connorshea connorshea closed this May 25, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment