Problems running aufs w/ linux-aws kernel

[niclic]

Problem description

Installing Docker on Ubuntu 16.04 LTS with the linux-aws kernel package fails because there is no corresponding linux-image-extra-$(uname -r) package for this kernel.

$ sudo apt-get install linux-image-extra-$(uname -r) \
>                      linux-image-extra-virtual
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package linux-image-extra-4.4.0-1012-aws
E: Couldn't find any package by glob 'linux-image-extra-4.4.0-1012-aws'
E: Couldn't find any package by regex 'linux-image-extra-4.4.0-1012-aws'

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:  Ubuntu 16.04.2 LTS
Release:  16.04
Codename: xenial

$ uname -r
4.4.0-1012-aws

This is similar to an issue described in #2252 (although presumably the mainline is not supported).

The docs seem to suggest that linux-image-extra-* packages are only required when installing on Trusty 14.04.

However, the aufs-driver docs include the same instructions, without reference to Ubuntu versions or code names.

This is confusing and probably needs clarification.

• What is the right way to install aufs support for Docker on Ubuntu 16.04?
• Are the linux-image-extra-$(uname -r) packages required to use aufs with Docker?

Problem location

URLS:

• https://docs.docker.com/engine/installation/linux/ubuntu/
• https://docs.docker.com/engine/userguide/storagedriver/aufs-driver/#configure-docker-with-aufs

Project version(s) affected

I was installing the latest version of docker ce on Ubuntu 16.04 LTS, but using the brand new linux-aws kernel package. This may affect other versions of Docker on Ubuntu.

Suggestions for a fix

Not sure if this is a suggestion (or even recommended), but here are my own findings on successfully installing on the linux-aws kernel.

The aufs-driver docs first suggest to check for aufs support:

$ grep aufs /proc/filesystems

nodev   aufs

Then to install the linux-image-extra-* packages:

sudo apt-get install linux-image-extra-$(uname -r) \
  linux-image-extra-virtual

Instead, I ran only this command :

sudo apt-get install linux-image-extra-virtual

This also installed linux-headers-4.4.0-72-generic, which presumably is what would be installed if I were not running the linux-aws kernel.

NOTE: At this point running grep aufs /proc/filesystems produces no output.

$ grep aufs /proc/filesystems

From here, I installed docker-ce as normal.

After installing Docker, running grep aufs /proc/filesystems produces the desired output:

$ grep aufs /proc/filesystems

nodev aufs

And after configuring Docker to use the aufs storage driver as described here, everything appears to run as expected.

$ sudo docker info

ubuntu@ip-10-150-100-85:~$ sudo docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 1
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 5
 Dirperm1 Supported: true
 ...

[niclic]

Still an issue with the latest linux-aws kernel package

$ uname -r
4.4.0-1013-aws

[niclic]

It sounds like this issue could be addressed by this fix: move aufs.ko from -extra to linux-image package, by removing the need to install any linux-image-extra-* packages.

Docker is ordinarily configured to use aufs, but the aufs kernel module is only available after installing the linux-image-extra package.

Please promote aufs.ko from the linux-image-extra package to the main linux-image package for all supported Ubuntu kernels and derivatives, so as to remove Docker's dependence on linux-image-extra.

This bug was fixed in the package linux - 4.4.0-75.96

I will follow up after verifying.

[heph]

Per the linux-aws_4.4.0-1009.18 changelog:

[config] AWS: aufs.ko moved to linux-image package

Confirmed on a test machine:

~# grep aufs /proc/filesystems
nodev	aufs
~# dpkg -l|grep linux-image
ii  linux-image-4.4.0-1013-aws          4.4.0-1013.22                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-1016-aws          4.4.0-1016.25                              amd64        Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-aws                     4.4.0.1016.19                              amd64        Linux kernel image for Amazon Web Services (AWS) systems.

[niclic]

Ah, thanks @heph. I have also confirmed your findings.

Seems like the fix has yet to make it to the offical Canonical Ubuntu 16.04 EC2 AMI.

After launching ami-80861296 (the most current release 20170414):

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:  Ubuntu 16.04.2 LTS
Release:  16.04
Codename: xenial

$ uname -r
4.4.0-1013-aws

$ grep aufs /proc/filesystems

$ dpkg -l | grep linux-image
ii  linux-image-4.4.0-1013-aws       4.4.0-1013.22  amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-aws                  4.4.0.1013.16  amd64 Linux kernel image for Amazon Web Services (AWS) systems.

I have to manually install the latest linux-image-aws package

$ sudo apt-get install linux-aws linux-headers-aws linux-image-aws -y

$ dpkg -l | grep linux-image
ii  linux-image-4.4.0-1013-aws       4.4.0-1013.22  amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-4.4.0-1016-aws       4.4.0-1016.25  amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
ii  linux-image-aws                  4.4.0.1016.19  amd64 Linux kernel image for Amazon Web Services (AWS) systems.

Then installing docker-ce as normal gives the desired result:

$ grep aufs /proc/filesystems
nodev aufs

Conclusions

• Going forward no additional packages should need to be installed to enable aufs support for Docker on Ubuntu.

• The docs should be updated to reflect this change.

BugReferences

• move aufs.ko from -extra to linux-image package
• aufs filesystem not available in linux-aws

[niclic]

The latest daily build has the most up to date linux-image-aws package, so I expect this to be available soon.

ubuntu/images-testing/hvm-ssd/ubuntu-xenial-daily-amd64-server-20170425 - ami-09d14b1f

Canonical, Ubuntu, None LTS, UNSUPPORTED daily amd64 xenial image build on 2017-04-25

After launching ami-09d14b1f, enabling aufs support with docker-ce has the desired effect, without any additional installation.

[niclic]

The latest official release is: Canonical, Ubuntu, 16.04 LTS, amd64 xenial image build on 2017-05-16

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:  Ubuntu 16.04.2 LTS
Release:  16.04
Codename: xenial

$ uname -r
4.4.0-1016-aws

After launching ami-20631a36 in us-east-1 and installing docker-ce, aufs support is enabled by default.

$ grep aufs /proc/filesystems
nodev aufs

No additional installations or configuration is required.

[tnguyenns1]

@niclic Thank you for your continuous updates!
From my understanding, this effects for Ubuntu 14.04 or newer as well, right?

If it's correct, i suppose the documentation is a bit out-of-date:

Recommended extra packages for Trusty 14.04
Unless you have a strong reason not to, install the linux-image-extra-* packages, which allow Docker to use the aufs storage drivers.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

/lifecycle locked