Compile fscache modules to speedup NFS shares

[paolomainardi]

This is needed in order to speedup NFS connections, as described here: http://chase-seibert.github.io/blog/2014/03/09/vagrant-cachefilesd.html

[SvenDowideit]

@tianon ? (I havn't tested, but I like the idea)

[paolomainardi]

This is just the kernel part, than we need the userspace client: http://linux.die.net/man/8/cachefilesd
As a static binary to be included in the ISO or a tcz package but needs to be created as it does not exist right now.

[tianon]

I'm +1 as long as it's compiled as a module and has negligible impact otherwise.

Can't the userspace client run in a container?

[paolomainardi]

thanks @tianon, let me check if the userspace client running in a container, very nice idea indeed, i'll back soon.

[paolomainardi]

A bit late, but i've finally find some spare time to test it.

Boot2docker built with fscache support (this PR):

root@boot2docker:/mnt/sda1/tmp# lsmod | grep cache
cachefiles             32768  0
fscache                36864  1 cachefiles

Cachefilesd dockerized Dockerfile:

FROM ubuntu:14.04
MAINTAINER Paolo Mainardi "paolo@twinbit.it"
ENV UPDATE_AT 1
RUN apt-get update && apt-get -y install cachefilesd
CMD /sbin/cachefilesd -n -f /etc/cachefilesd.conf -s

RUN output:

% docker run --privileged -it --rm cachefilesd                                                                                                                                         
About to bind cache
CacheFiles bind failed: errno 95 (Operation not supported)

Container dmesg:

CacheFiles: Security denies permission to nominate security context: error -95
CacheFiles: Security denies permission to nominate security context: error -95
CacheFiles: Security denies permission to nominate security context: error -95

The problem seems related to this function: https://github.com/dz0ny/nims-kernel-ics/blob/master/fs/cachefiles/security.c#L34

I'm still trying to understand what is going on, in the meanwhile if have any suggestions they are more than welcome.

[paolomainardi]

Ok, it seems that cachefilesd client requires some selinux kernel configuration, i'm going to put everything together, i'll keep you posted.

[paolomainardi]

Updated with required kernel configurations.
I've also tried to run cachefilesd in a container and it works as expected.

[paolomainardi]

@tianon any updates ?

[paolomainardi]

Sorry for bumping up, any updates @tianon @SvenDowideit ?

[SvenDowideit]

adding SELinux brings up red flags which mean we need to do some more research - we discussed it somewhere, and were told there were consequences - but this might be under slightly different details.

[tianon]

Indeed, that discussion was in #459 (comment).

[paolomainardi]

@tianon @SvenDowideit i've tried extensively in the meanwhile without any nasty side effects. Anyway SELINUX is disabled by default:

+CONFIG_SECURITY_SELINUX_BOOTPARAM=y
+CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=0
+CONFIG_SECURITY_SELINUX_DISABLE=y

[SvenDowideit]

@paolomainardi can you rebase this PR please?

[paolomainardi]

@SvenDowideit yes, i'll do it ASAP.

[paolomainardi]

Rebased, @SvenDowideit everything works as expected.

In order to test it, follows this steps.

0. Rebuild the boot2docker.iso using this kernel_config.

1. Configure NFS on OSX side.

Edit /etc/exports as follows:

/Users -mapall=YOURUSER:staff BOOT2DOCKER_IP

Replacing YOURUSER with the result of whoami and BOOT2DOCKER_IP with the result of boot2docker ip

Now restart NFS service: sudo nfsd stop && sudo nfsd start

2. Create cachefilesd container and load the kernel module:

FROM ubuntu:14.04
MAINTAINER Paolo Mainardi "paolo@twinbit.it"
ENV UPDATE_AT 1
RUN apt-get update && apt-get -y install cachefilesd
CMD /sbin/cachefilesd -n -f /etc/cachefilesd.conf -s

Compile and run:

boot2docker ssh "sudo modprobe cachefiles"
docker build -t cachefilesd .
docker run --privileged -d -v /mnt/sda1/var/cache/fscache:/var/cache/fscache cachefilesd

You should see something like this:

% docker ps                                                                                                                                                                              
CONTAINER ID        IMAGE                               COMMAND                CREATED              STATUS              PORTS                    NAMES
5cc4416c9f58        cachefilesd:latest                  "/bin/sh -c '/sbin/c   13 minutes ago       Up 13 minutes                                lonely_goldstine

3. Unmount vboxfs and mount NFS with fsc attribute on:

boot2docker ssh 'sudo umount /Users'
boot2docker ssh 'sudo /usr/local/etc/init.d/nfs-client start'
boot2docker ssh 'sudo mount.nfs BOOT2DOCKER_IP:/Users /Users -o fsc,rw,async,noatime,rsize=32768,wsize=32768,proto=tcp'

Where BOOT2DOCKER_IP is the result of boot2docker ip.

Check if it works correctly, you should see something like this:

% boot2docker ssh 'sudo cat /proc/fs/nfsfs/volumes'                                                                                                                             
NV SERVER   PORT DEV     FSID              FSC
v3 c0a83b03  801 0:72    1000003:0         yes

% boot2docker ssh 'sudo du -chs /mnt/sda1/var/cache/fscache'
16.3M   /mnt/sda1/var/cache/fscache
16.3M   total

[SvenDowideit]

I'll do some testing in the next week - doing some conference speaking atm

[paolomainardi]

ok @SvenDowideit thanks

[paolomainardi]

related to #828 #817 #224

[SvenDowideit]

@tianon I can't find a downside to this change.

[paolomainardi]

Thanks for testing @SvenDowideit

[tianon]

Thanks @SvenDowideit ❤️

[tianon]

LGTM

[SvenDowideit]

i finally made time to play with the experimental volume plugins -

docker run --rm -it --volume-driver=nfs -v 127.0.0.1/data:/no busybox ls -la works :)

see https://github.com/SvenDowideit/docker-volumes-nfs for where i'm heading.

[thaJeztah]

@SvenDowideit that's interesting! Looks like your README is incomplete

To use the plugin when mounting a volume, run:
License
MIT

I tried, but 😉 :

root@e0b10a13dcf8:/# License MIT
bash: License: command not found

[paolomainardi]

Wow, thanks a lot guys!

Now i nave to test new NFS volume driver and we need a simple way to choose vboxfs/NFS share.

[SvenDowideit]

@thaJeztah ooops - it was in the Makefile, but i forgot to talk it up :)

I havn't had time since my flight, but hopefully I'll find time during one of the hackathons (or worst case, July, when I'm back in AU.

wrt a simple way to choose the share - yup, that too is coming :)