Apache -D foreground fails

[wernermaes]

When I start the docker container I get:

Module auth_kerb disabled.
To activate the new configuration, you need to run:
service apache2 restart
Module ssl already disabled
Enabling module rewrite.
To activate the new configuration, you need to run:
service apache2 restart
Stopping web server: apache2.
AH00534: apache2: Configuration error: No MPM loaded.
Action '-D FOREGROUND' failed.
The Apache error log may have more information.

Any ideas?

[kwk]

Please start with the example setup directory.

[jovandeginste]

So after a lot of testing, I came to this conclusion:

The problem arises because of a2dismod auth_kerb - this tries to remove the auth_kerb.load files from mods-enabled. We run on centos7 with overlay+xfs (proof-of-concept)

The core issue is docker layers removing things from earlier layers, but this not being reflected properly.

In this case, Apache starts, still sees a file "auth_kerb.load" (because of the bug), but can't access the data. There it stops loading it's config files.

If you run the image with bash, after you run /root/start-apache.sh manually, you see this:

# ls /etc/apache2/mods-enabled/ -l
ls: cannot access /etc/apache2/mods-enabled/auth_kerb.load: No such file or directory
total 0
lrwxrwxrwx 1 root root 36 Jan 20 14:13 access_compat.load -> ../mods-available/access_compat.load
lrwxrwxrwx 1 root root 28 Jan 20 14:13 alias.conf -> ../mods-available/alias.conf
lrwxrwxrwx 1 root root 28 Jan 20 14:13 alias.load -> ../mods-available/alias.load
lrwxrwxrwx 1 root root 33 Jan 20 14:13 auth_basic.load -> ../mods-available/auth_basic.load
?????????? ? ?    ?     ?            ? auth_kerb.load
lrwxrwxrwx 1 root root 33 Jan 20 14:13 authn_core.load -> ../mods-available/authn_core.load
lrwxrwxrwx 1 root root 33 Jan 20 14:13 authn_file.load -> ../mods-available/authn_file.load
</snip>

For reference concerning storage drivers: moby/moby#18298

[jbehrends]

I just ran into this very same issue on CentOS 7.2 + xfs + overlay filesystem

After reading this:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

I found that our xfs filesystem was not created with the mkfs flag of "ftype=1"

Since we didn't want to nuke the whole box/filesystem we decided to create a mount for /var/lib/docker using this flag and the issue is now resolved.

[nemonik]

@jbehrends how'd you create that mount for /var/lib/docker?

[nemonik]

So does anyone know how to create a mount using this flag to resolve the issue on CentOS 7.2?

[jovandeginste]

@nemonik does this not work: http://man7.org/linux/man-pages/man8/mkfs.xfs.8.html ?

[zoza1982]

Thanks guys for pointing on to this bug.
@wernermaes @nemonik @jovandeginste I had the same problem with the same OS and I did this with success:

1. stop all containers
2. stop docker service service docker stop
3. create a 10g file for hosting newly formatted xfs. dd if=/dev/zero of=docker_fs bs=1G count=10 ( change "count" to whatever space you want to give to docker overlay )
4. format the file with as filesystem with proper flag mkfs -t xfs -n ftype=1 docker_fs
5. mount the filesystem via loop mount -o loop registry_fs /var/lib/docker/
6. add that into /etc/fstab so it persist boot /root/registry_fs /var/lib/docker xfs rw,loop,auto 0 0
7. start back up docker service service docker start
8. start back up your docker registry v2 and docker-registry-frontend
9. problem solved ( at least for me )

Once you are done, you should see these 3 lines when doing df -h
/dev/loop1 9.8G 726M 8.5G 8% /var/lib/docker overlay 9.8G 726M 8.5G 8% /var/lib/docker/overlay/354ca110acd8eb1f7e1ccee2399e30e3120fc868a313d74c1aceef9ea552ad46/merged shm 64M 0 64M 0% /var/lib/docker/containers/821eb4b58dc411605441f3ea539ac03f933b8152792f5157484e91bbf06c9f49/shm

[guydavis]

For those on AWS, here's what I did to workaround this issue using Terraform in your aws_instance resource:

ebs_block_device  {
  device_name = "/dev/xvdg"
  volume_type = "gp2"
  volume_size = "25"
  delete_on_termination = true
}

Then execute this script to create the new mount:

sudo lsblk # List available volumes
sudo mkfs.xfs -n ftype=1 /dev/xvdg
sudo file -s /dev/xvdg  # Show file type of newly created mount
export UUID=`ls -al /dev/disk/by-uuid/ | grep xvdg | tr -s ' ' | cut -d ' ' -f 9`
sudo tee -a /etc/fstab <<EOF  
UUID=$UUID /mnt/data		  xfs	  defaults,nofail        0 2
EOF
sudo mkdir /mnt/data
sudo mount /mnt/data
df -h  # Show the newly created disk mount
sudo mv /var/lib/docker /mnt/data/ 
sudo ln -s /mnt/data/docker /var/lib/docker
sudo systemctl enable docker
sudo systemctl start docker

Applies to CentOS 7 and Docker 1.11, which I can't upgrade due to other dependencies right now.