CrashLoopBackOff on the database

[MaxenceAdnot]

I'm trying to deploy the deis-dev chart on AWS and it seems that the database refuses to start...

CoreOS version is 976.0.0 (alpha)

docker logs 6ece7bde9d :

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data ...

I also noticed that line in kernel logs :

Mar 12 21:07:48 ip-10-0-0-144.eu-west-1.compute.internal dockerd[1128]: time="2016-03-12T21:07:48.916811751Z" level=warning msg="DEPRECATED: Setting host configuration options when the container starts is deprecated and will be removed in
Mar 12 21:07:48 ip-10-0-0-144.eu-west-1.compute.internal systemd[1]: Started docker container 6ece7bde9d462a3a1d66d480a9e9aa321e895f61c8dd6f7e30f0fa14a7e9fd72.
Mar 12 21:07:49 ip-10-0-0-144.eu-west-1.compute.internal kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Mar 12 21:07:49 ip-10-0-0-144.eu-west-1.compute.internal systemd[1]: Stopped docker container 6ece7bde9d462a3a1d66d480a9e9aa321e895f61c8dd6f7e30f0fa14a7e9fd72.

I don't really like this SELinux message :/

Any ideas ?

[slack]

We spoke a bit on the new Deis slack! (Shameless plug, join us at https://slack.deis.io)

I was able to reproduce on a CoreOS 983.0.0 cluster out of the box. May be related to userns + selinux + Docker 1.10.2 but I haven't pinned down a specific change just yet.

I was able to successfully boot the postgres component by specifying a volume, mounted at /var/lib/postgres.

@MaxenceAdnot if you could modify the deis-database-rc.yaml with the changes in https://github.com/deis/charts/pull/160/files that would be awesome.

That can be accomplished by:

1. helm uninstall deis-dev <-- will remove your deis install
2. helm edit deis-dev <-- hand edit tpl/deis-database-rc.yaml
3. run helm generate deis-dev
4. helm install deis-dev

OR:

1. kubectl --namespace=deis edit rc deis-database <-- hand-edit the volume information from the PR
2. kubectl --namespace=deis delete deis-databse-XYZ123 < --- delete the database pod

Edit: clarity

[MaxenceAdnot]

Thank you for the quick fix. I will try this in a few hours and let you know.

[MaxenceAdnot]

Your patch is working fine ! Thank you @slack

[slack]

Excellent, thanks for testing! Will get this merged in the morning.

[bacongobbler]

The upstream issue for this is moby/moby#7952. It seems to be Docker + btrfs with SELinux enabled causes this issue. Can you try running docker on a non-btrfs backend and see if you see the same issue?

This did not necessarily resolve the issue, just circumvent it. Volume mounts are likely non-btrfs so SELinux is playing nicely with it. This project is also not designed around being backed by a persistent disk as demontrated in the end-to-end tests at https://ci.deis.io/.

You'll likely see this issue again in another form in the future, so I don't feel this problem is resolved. I'm going to revert this change and further debug this issue.

[slack]

core@ip-10-0-0-175 ~ $ docker info
Containers: 43
 Running: 7
 Paused: 0
 Stopped: 36
Images: 10
Server Version: 1.10.2
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.4-coreos
Operating System: CoreOS 983.0.0 (Coeur Rouge)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.679 GiB
Name: ip-10-0-0-175.us-west-2.compute.internal
ID: OP6Q:UNHM:BTT4:FCYX:3QTL:HZUQ:ET7E:EOXC:RYJO:L5ZK:VLQO:YYCH

[MaxenceAdnot]

I don't think CoreOS is using btrfs at all (from my memory Docker on CoreOS is using overlay as storage driver).

[bacongobbler]

Some concrete evidence supporting this would be this comment: moby/moby#7952 (comment)

@slack I think this includes overlayfs as well, judging from that comment:

Kernel engineers are working on a fix for this and potentially Overlayfs if it gets merged into the container.

[bacongobbler]

Reading further down the page, the temporary fix seems to be removing --selinux-enabled from the docker daemon's option flag list. Overlay is indeed broken as well.

Source for fix: moby/moby#7952 (comment)