Question: does Docker / LXC de-duplicate memory? If not, is it feasible?

[jokeyrhyme]

Had an interesting question posed by a coworker today: if multiple containers make use of the same shared libraries, do they occupy multiple copies in memory?

I know that de-duplication can be done at the file-system level if you use Btrfs, etc. Does the Linux kernel have such a feature for memory? If so, to what degree is this feature in operation when similar containers are in use?

[jokeyrhyme]

I did just find this: http://en.wikipedia.org/wiki/Kernel_SamePage_Merging_(KSM)
Does this impact the answer in any way?

[jokeyrhyme]

From #docker @ irc.freenode.net:

jokeyrhyme Is memory de-duplicated in a similar way to the file-system when you have multiple containers using the same shared libraries?
erikh no
erikh shared libraries on a single host have some interesting memory properties but not between containers. containers, almost by definition, are independent and isolated.
jokeyrhyme Hmmm. So if you have 3 containers all based on the same base image (e.g. Ubuntu) then you potentially have 3 copies of glibc and other shared libraries in the host's RAM?
erikh jokeyrhyme: yep
jokeyrhyme I'm guessing that would require work within the Linux kernel to de-duplicate memory?
erikh probably a lot more expensive than it'd be worth, especially on anything that isn't embedded class
jokeyrhyme Hmmm, I just found this: http://en.wikipedia.org/wiki/Kernel_SamePage_Merging_(KSM)
erikh we're talking about kilobytes or megabytes here for the shared libraries in a world where getting a server with 64GB of ram is almost consumer class
jokeyrhyme What if you had 500 Apache HTTP + mod_php containers on the same machine? Would there not be room for significant memory efficiency if shared libraries were deduplicated?
erikh from shared libraries? as opposed to php?

[jokeyrhyme]

More from #docker @ irc.freenode.net:

jokeyrhyme All of the above?
SvenDowideit jokeyrhyme: as far as i grok, yes, the linux kernel does memory de-duplication on the namespaced binary
erikh SvenDowideit: really?
erikh on shared libs?
SvenDowideit thats what i read somewhere - :/
SvenDowideit need to re-read and actually do some reproducable tests tho
erikh fair enough. I just presumed that was madness
SvenDowideit here's a horror thing - i actually know that in windows, its known if a page is data or sharedlibrary
SvenDowideit and shared library pages are marked as such, and writing to them causes a page fault
SvenDowideit i assumed that linux is just as clever, but i'm probably not going to know that til next year, considering where i am timewise
erikh it looks like KSM here has a significant perf cost
erikh I can't see why deduping memory would be the default in the kernel, but linux has surprised me before
erikh anyhow I need to go do things
SvenDowideit wereas i assumed that shared libraries would be central to things - and happen shared before duping
SvenDowideit confirms nothing, and raises more questions :)

[jokeyrhyme]

@coaic @danzy @benbarclay

[crosbymichael]

Each container has it's own IPC namespace so certain things are not shared between the container.

See: http://lwn.net/Articles/187274/ and http://man7.org/linux/man-pages/man7/svipc.7.html as it applies to containers in their own IPC namespaces.

[crosbymichael]

We can probably move this conversation to the mailing list as we try to keep the github issues for issues.

[AkihiroSuda]

So actually deduplication also works for containers, right?

When only 1 container is running, only 4 pages are in the Shared_Clean state.
However, when running 2 containers, 2792 pages enter the Shared_Clean state.

$ docker run -itd --name c1 ubuntu bash
$ sudo pmap -XX $(docker inspect --format '{{.State.Pid}}' c1) 
2222:   bash
         Address Perm   Offset Device    Inode  Size  Rss  Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked                   VmFlagsMapping
        00400000 r-xp 00000000  08:03 23466631   956  896  896            0            0           896             0        896         0             0    0              4           4      0    rd ex mr mw me dw sd  bash
        006ef000 r--p 000ef000  08:03 23466631     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd mr mw me dw ac sd  bash
        006f0000 rw-p 000f0000  08:03 23466631    36   36   36            0            0             0            36         36        36             0    0              4           4      0 rd wr mr mw me dw ac sd  bash
        006f9000 rw-p 00000000  00:00        0    24   24   24            0            0             0            24         24        24             0    0              4           4      0    rd wr mr mw me ac sd  
        02649000 rw-p 00000000  00:00        0   256  256  256            0            0             0           256        256       256             0    0              4           4      0    rd wr mr mw me ac sd  [heap]
    7fd8a7550000 r-xp 00000000  08:03 23862337    44   44   44            0            0            44             0         44         0             0    0              4           4      0       rd ex mr mw me sd  libnss_files-2.19.so
    7fd8a755b000 ---p 0000b000  08:03 23862337  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_files-2.19.so
    7fd8a775a000 r--p 0000a000  08:03 23862337     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_files-2.19.so
    7fd8a775b000 rw-p 0000b000  08:03 23862337     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_files-2.19.so
    7fd8a775c000 r-xp 00000000  08:03 23862341    44   44   44            0            0            44             0         44         0             0    0              4           4      0       rd ex mr mw me sd  libnss_nis-2.19.so
    7fd8a7767000 ---p 0000b000  08:03 23862341  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_nis-2.19.so
    7fd8a7966000 r--p 0000a000  08:03 23862341     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_nis-2.19.so
    7fd8a7967000 rw-p 0000b000  08:03 23862341     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_nis-2.19.so
    7fd8a7968000 r-xp 00000000  08:03 23862331    92   64   64            0            0            64             0         64         0             0    0              4           4      0       rd ex mr mw me sd  libnsl-2.19.so
    7fd8a797f000 ---p 00017000  08:03 23862331  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnsl-2.19.so
    7fd8a7b7e000 r--p 00016000  08:03 23862331     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnsl-2.19.so
    7fd8a7b7f000 rw-p 00017000  08:03 23862331     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnsl-2.19.so
    7fd8a7b80000 rw-p 00000000  00:00        0     8    0    0            0            0             0             0          0         0             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a7b82000 r-xp 00000000  08:03 23862333    36   36   36            0            0            36             0         36         0             0    0              4           4      0       rd ex mr mw me sd  libnss_compat-2.19.so
    7fd8a7b8b000 ---p 00009000  08:03 23862333  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_compat-2.19.so
    7fd8a7d8a000 r--p 00008000  08:03 23862333     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_compat-2.19.so
    7fd8a7d8b000 rw-p 00009000  08:03 23862333     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_compat-2.19.so
    7fd8a7d8c000 r-xp 00000000  08:03 23862203  1772 1424 1424            0            0          1424             0       1424         0             0    0              4           4      0       rd ex mr mw me sd  libc-2.19.so
    7fd8a7f47000 ---p 001bb000  08:03 23862203  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libc-2.19.so
    7fd8a8146000 r--p 001ba000  08:03 23862203    16   16   16            0            0             0            16         16        16             0    0              4           4      0       rd mr mw me ac sd  libc-2.19.so
    7fd8a814a000 rw-p 001be000  08:03 23862203     8    8    8            0            0             0             8          8         8             0    0              4           4      0    rd wr mr mw me ac sd  libc-2.19.so
    7fd8a814c000 rw-p 00000000  00:00        0    20   16   16            0            0             0            16         16        16             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a8151000 r-xp 00000000  08:03 23862219    12   12   12            0            0            12             0         12         0             0    0              4           4      0       rd ex mr mw me sd  libdl-2.19.so
    7fd8a8154000 ---p 00003000  08:03 23862219  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libdl-2.19.so
    7fd8a8353000 r--p 00002000  08:03 23862219     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libdl-2.19.so
    7fd8a8354000 rw-p 00003000  08:03 23862219     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libdl-2.19.so
    7fd8a8355000 r-xp 00000000  08:03 23862386   148  148  148            0            0           148             0        148         0             0    0              4           4      0       rd ex mr mw me sd  libtinfo.so.5.9
    7fd8a837a000 ---p 00025000  08:03 23862386  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libtinfo.so.5.9
    7fd8a8579000 r--p 00024000  08:03 23862386    16   16   16            0            0             0            16         16        16             0    0              4           4      0       rd mr mw me ac sd  libtinfo.so.5.9
    7fd8a857d000 rw-p 00028000  08:03 23862386     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libtinfo.so.5.9
    7fd8a857e000 r-xp 00000000  08:03 23862181   140  136  136            0            0           136             0        136         0             0    0              4           4      0    rd ex mr mw me dw sd  ld-2.19.so
    7fd8a8798000 rw-p 00000000  00:00        0    12   12   12            0            0             0            12         12        12             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a879e000 rw-p 00000000  00:00        0     8    8    8            0            0             0             8          8         8             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a87a0000 r--p 00022000  08:03 23862181     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd mr mw me dw ac sd  ld-2.19.so
    7fd8a87a1000 rw-p 00023000  08:03 23862181     4    4    4            0            0             0             4          4         4             0    0              4           4      0 rd wr mr mw me dw ac sd  ld-2.19.so
    7fd8a87a2000 rw-p 00000000  00:00        0     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  
    7ffdf9819000 rw-p 00000000  00:00        0   136   28   28            0            0             0            28         28        28             0    0              4           4      0    rd wr mr mw me gd ac  [stack]
    7ffdf9863000 r--p 00000000  00:00        0     8    0    0            0            0             0             0          0         0             0    0              4           4      0    rd mr pf io de dd sd  [vvar]
    7ffdf9865000 r-xp 00000000  00:00        0     8    4    0            4            0             0             0          4         0             0    0              4           4      0    rd ex mr mw me de sd  [vdso]
ffffffffff600000 r-xp 00000000  00:00        0     4    0    0            0            0             0             0          0         0             0    0              4           4      0                   rd ex  [vsyscall]
                                               ===== ==== ==== ============ ============ ============= ============= ========== ========= ============= ==== ============== =========== ====== 
                                               18172 3288 3284            4            0          2804           480       3288       480             0    0            180         180      0 KB 

$ docker run -itd --name c2 ubuntu bash
$ sudo pmap -XX $(docker inspect --format '{{.State.Pid}}' c1) 
2222:   bash
         Address Perm   Offset Device    Inode  Size  Rss  Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked                   VmFlagsMapping
        00400000 r-xp 00000000  08:03 23466631   956  896  448          896            0             0             0        896         0             0    0              4           4      0    rd ex mr mw me dw sd  bash
        006ef000 r--p 000ef000  08:03 23466631     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd mr mw me dw ac sd  bash
        006f0000 rw-p 000f0000  08:03 23466631    36   36   36            0            0             0            36         36        36             0    0              4           4      0 rd wr mr mw me dw ac sd  bash
        006f9000 rw-p 00000000  00:00        0    24   24   24            0            0             0            24         24        24             0    0              4           4      0    rd wr mr mw me ac sd  
        02649000 rw-p 00000000  00:00        0   256  256  256            0            0             0           256        256       256             0    0              4           4      0    rd wr mr mw me ac sd  [heap]
    7fd8a7550000 r-xp 00000000  08:03 23862337    44   44   22           44            0             0             0         44         0             0    0              4           4      0       rd ex mr mw me sd  libnss_files-2.19.so
    7fd8a755b000 ---p 0000b000  08:03 23862337  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_files-2.19.so
    7fd8a775a000 r--p 0000a000  08:03 23862337     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_files-2.19.so
    7fd8a775b000 rw-p 0000b000  08:03 23862337     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_files-2.19.so
    7fd8a775c000 r-xp 00000000  08:03 23862341    44   44   22           44            0             0             0         44         0             0    0              4           4      0       rd ex mr mw me sd  libnss_nis-2.19.so
    7fd8a7767000 ---p 0000b000  08:03 23862341  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_nis-2.19.so
    7fd8a7966000 r--p 0000a000  08:03 23862341     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_nis-2.19.so
    7fd8a7967000 rw-p 0000b000  08:03 23862341     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_nis-2.19.so
    7fd8a7968000 r-xp 00000000  08:03 23862331    92   64   32           64            0             0             0         64         0             0    0              4           4      0       rd ex mr mw me sd  libnsl-2.19.so
    7fd8a797f000 ---p 00017000  08:03 23862331  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnsl-2.19.so
    7fd8a7b7e000 r--p 00016000  08:03 23862331     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnsl-2.19.so
    7fd8a7b7f000 rw-p 00017000  08:03 23862331     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnsl-2.19.so
    7fd8a7b80000 rw-p 00000000  00:00        0     8    0    0            0            0             0             0          0         0             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a7b82000 r-xp 00000000  08:03 23862333    36   36   18           36            0             0             0         36         0             0    0              4           4      0       rd ex mr mw me sd  libnss_compat-2.19.so
    7fd8a7b8b000 ---p 00009000  08:03 23862333  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_compat-2.19.so
    7fd8a7d8a000 r--p 00008000  08:03 23862333     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_compat-2.19.so
    7fd8a7d8b000 rw-p 00009000  08:03 23862333     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_compat-2.19.so
    7fd8a7d8c000 r-xp 00000000  08:03 23862203  1772 1424  718         1412            0            12             0       1424         0             0    0              4           4      0       rd ex mr mw me sd  libc-2.19.so
    7fd8a7f47000 ---p 001bb000  08:03 23862203  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libc-2.19.so
    7fd8a8146000 r--p 001ba000  08:03 23862203    16   16   16            0            0             0            16         16        16             0    0              4           4      0       rd mr mw me ac sd  libc-2.19.so
    7fd8a814a000 rw-p 001be000  08:03 23862203     8    8    8            0            0             0             8          8         8             0    0              4           4      0    rd wr mr mw me ac sd  libc-2.19.so
    7fd8a814c000 rw-p 00000000  00:00        0    20   16   16            0            0             0            16         16        16             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a8151000 r-xp 00000000  08:03 23862219    12   12    6           12            0             0             0         12         0             0    0              4           4      0       rd ex mr mw me sd  libdl-2.19.so
    7fd8a8154000 ---p 00003000  08:03 23862219  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libdl-2.19.so
    7fd8a8353000 r--p 00002000  08:03 23862219     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libdl-2.19.so
    7fd8a8354000 rw-p 00003000  08:03 23862219     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libdl-2.19.so
    7fd8a8355000 r-xp 00000000  08:03 23862386   148  148   74          148            0             0             0        148         0             0    0              4           4      0       rd ex mr mw me sd  libtinfo.so.5.9
    7fd8a837a000 ---p 00025000  08:03 23862386  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libtinfo.so.5.9
    7fd8a8579000 r--p 00024000  08:03 23862386    16   16   16            0            0             0            16         16        16             0    0              4           4      0       rd mr mw me ac sd  libtinfo.so.5.9
    7fd8a857d000 rw-p 00028000  08:03 23862386     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libtinfo.so.5.9
    7fd8a857e000 r-xp 00000000  08:03 23862181   140  136   70          132            0             4             0        136         0             0    0              4           4      0    rd ex mr mw me dw sd  ld-2.19.so
    7fd8a8798000 rw-p 00000000  00:00        0    12   12   12            0            0             0            12         12        12             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a879e000 rw-p 00000000  00:00        0     8    8    8            0            0             0             8          8         8             0    0              4           4      0    rd wr mr mw me ac sd  
    7fd8a87a0000 r--p 00022000  08:03 23862181     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd mr mw me dw ac sd  ld-2.19.so
    7fd8a87a1000 rw-p 00023000  08:03 23862181     4    4    4            0            0             0             4          4         4             0    0              4           4      0 rd wr mr mw me dw ac sd  ld-2.19.so
    7fd8a87a2000 rw-p 00000000  00:00        0     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  
    7ffdf9819000 rw-p 00000000  00:00        0   136   28   28            0            0             0            28         28        28             0    0              4           4      0    rd wr mr mw me gd ac  [stack]
    7ffdf9863000 r--p 00000000  00:00        0     8    0    0            0            0             0             0          0         0             0    0              4           4      0    rd mr pf io de dd sd  [vvar]
    7ffdf9865000 r-xp 00000000  00:00        0     8    4    0            4            0             0             0          4         0             0    0              4           4      0    rd ex mr mw me de sd  [vdso]
ffffffffff600000 r-xp 00000000  00:00        0     4    0    0            0            0             0             0          0         0             0    0              4           4      0                   rd ex  [vsyscall]
                                               ===== ==== ==== ============ ============ ============= ============= ========== ========= ============= ==== ============== =========== ====== 
                                               18172 3288 1890         2792            0            16           480       3288       480             0    0            180         180      0 KB 
$ sudo pmap -XX $(docker inspect --format '{{.State.Pid}}' c2)                                                                         
2343:   bash
         Address Perm   Offset Device    Inode  Size  Rss  Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked                   VmFlagsMapping
        00400000 r-xp 00000000  08:03 23466631   956  896  448          896            0             0             0        896         0             0    0              4           4      0    rd ex mr mw me dw sd  bash
        006ef000 r--p 000ef000  08:03 23466631     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd mr mw me dw ac sd  bash
        006f0000 rw-p 000f0000  08:03 23466631    36   36   36            0            0             0            36         36        36             0    0              4           4      0 rd wr mr mw me dw ac sd  bash
        006f9000 rw-p 00000000  00:00        0    24   24   24            0            0             0            24         24        24             0    0              4           4      0    rd wr mr mw me ac sd  
        00a16000 rw-p 00000000  00:00        0   256  256  256            0            0             0           256        256       256             0    0              4           4      0    rd wr mr mw me ac sd  [heap]
    7f6a3cca1000 r-xp 00000000  08:03 23862337    44   44   22           44            0             0             0         44         0             0    0              4           4      0       rd ex mr mw me sd  libnss_files-2.19.so
    7f6a3ccac000 ---p 0000b000  08:03 23862337  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_files-2.19.so
    7f6a3ceab000 r--p 0000a000  08:03 23862337     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_files-2.19.so
    7f6a3ceac000 rw-p 0000b000  08:03 23862337     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_files-2.19.so
    7f6a3cead000 r-xp 00000000  08:03 23862341    44   44   22           44            0             0             0         44         0             0    0              4           4      0       rd ex mr mw me sd  libnss_nis-2.19.so
    7f6a3ceb8000 ---p 0000b000  08:03 23862341  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_nis-2.19.so
    7f6a3d0b7000 r--p 0000a000  08:03 23862341     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_nis-2.19.so
    7f6a3d0b8000 rw-p 0000b000  08:03 23862341     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_nis-2.19.so
    7f6a3d0b9000 r-xp 00000000  08:03 23862331    92   64   32           64            0             0             0         64         0             0    0              4           4      0       rd ex mr mw me sd  libnsl-2.19.so
    7f6a3d0d0000 ---p 00017000  08:03 23862331  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnsl-2.19.so
    7f6a3d2cf000 r--p 00016000  08:03 23862331     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnsl-2.19.so
    7f6a3d2d0000 rw-p 00017000  08:03 23862331     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnsl-2.19.so
    7f6a3d2d1000 rw-p 00000000  00:00        0     8    0    0            0            0             0             0          0         0             0    0              4           4      0    rd wr mr mw me ac sd  
    7f6a3d2d3000 r-xp 00000000  08:03 23862333    36   36   18           36            0             0             0         36         0             0    0              4           4      0       rd ex mr mw me sd  libnss_compat-2.19.so
    7f6a3d2dc000 ---p 00009000  08:03 23862333  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libnss_compat-2.19.so
    7f6a3d4db000 r--p 00008000  08:03 23862333     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libnss_compat-2.19.so
    7f6a3d4dc000 rw-p 00009000  08:03 23862333     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libnss_compat-2.19.so
    7f6a3d4dd000 r-xp 00000000  08:03 23862203  1772 1420  714         1412            0             8             0       1420         0             0    0              4           4      0       rd ex mr mw me sd  libc-2.19.so
    7f6a3d698000 ---p 001bb000  08:03 23862203  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libc-2.19.so
    7f6a3d897000 r--p 001ba000  08:03 23862203    16   16   16            0            0             0            16         16        16             0    0              4           4      0       rd mr mw me ac sd  libc-2.19.so
    7f6a3d89b000 rw-p 001be000  08:03 23862203     8    8    8            0            0             0             8          8         8             0    0              4           4      0    rd wr mr mw me ac sd  libc-2.19.so
    7f6a3d89d000 rw-p 00000000  00:00        0    20   16   16            0            0             0            16         16        16             0    0              4           4      0    rd wr mr mw me ac sd  
    7f6a3d8a2000 r-xp 00000000  08:03 23862219    12   12    6           12            0             0             0         12         0             0    0              4           4      0       rd ex mr mw me sd  libdl-2.19.so
    7f6a3d8a5000 ---p 00003000  08:03 23862219  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libdl-2.19.so
    7f6a3daa4000 r--p 00002000  08:03 23862219     4    4    4            0            0             0             4          4         4             0    0              4           4      0       rd mr mw me ac sd  libdl-2.19.so
    7f6a3daa5000 rw-p 00003000  08:03 23862219     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libdl-2.19.so
    7f6a3daa6000 r-xp 00000000  08:03 23862386   148  148   74          148            0             0             0        148         0             0    0              4           4      0       rd ex mr mw me sd  libtinfo.so.5.9
    7f6a3dacb000 ---p 00025000  08:03 23862386  2044    0    0            0            0             0             0          0         0             0    0              4           4      0             mr mw me sd  libtinfo.so.5.9
    7f6a3dcca000 r--p 00024000  08:03 23862386    16   16   16            0            0             0            16         16        16             0    0              4           4      0       rd mr mw me ac sd  libtinfo.so.5.9
    7f6a3dcce000 rw-p 00028000  08:03 23862386     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  libtinfo.so.5.9
    7f6a3dccf000 r-xp 00000000  08:03 23862181   140  132   66          132            0             0             0        132         0             0    0              4           4      0    rd ex mr mw me dw sd  ld-2.19.so
    7f6a3dee9000 rw-p 00000000  00:00        0    12   12   12            0            0             0            12         12        12             0    0              4           4      0    rd wr mr mw me ac sd  
    7f6a3deef000 rw-p 00000000  00:00        0     8    8    8            0            0             0             8          8         8             0    0              4           4      0    rd wr mr mw me ac sd  
    7f6a3def1000 r--p 00022000  08:03 23862181     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd mr mw me dw ac sd  ld-2.19.so
    7f6a3def2000 rw-p 00023000  08:03 23862181     4    4    4            0            0             0             4          4         4             0    0              4           4      0 rd wr mr mw me dw ac sd  ld-2.19.so
    7f6a3def3000 rw-p 00000000  00:00        0     4    4    4            0            0             0             4          4         4             0    0              4           4      0    rd wr mr mw me ac sd  
    7ffcbe796000 rw-p 00000000  00:00        0   136   24   24            0            0             0            24         24        24             0    0              4           4      0    rd wr mr mw me gd ac  [stack]
    7ffcbe7dd000 r--p 00000000  00:00        0     8    0    0            0            0             0             0          0         0             0    0              4           4      0    rd mr pf io de dd sd  [vvar]
    7ffcbe7df000 r-xp 00000000  00:00        0     8    4    0            4            0             0             0          4         0             0    0              4           4      0    rd ex mr mw me de sd  [vdso]
ffffffffff600000 r-xp 00000000  00:00        0     4    0    0            0            0             0             0          0         0             0    0              4           4      0                   rd ex  [vsyscall]
                                               ===== ==== ==== ============ ============ ============= ============= ========== ========= ============= ==== ============== =========== ====== 
                                               18172 3276 1878         2792            0             8           476       3276       476             0    0            180         180      0 KB 

$ docker info
Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.10.0-rc3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.2.0-25-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.6 GiB

[AkihiroSuda]

SELF-RESOLVED, deduplication works for aufs and overlayfs, but not work for devicemapper or btrfs.
http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384

Sorry for bothering.

[thaJeztah]

@AkihiroSuda do you think we should mention it (or mention it more explicitly) in our storage driver section? https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

[AkihiroSuda]

It seems already mentioned 😄

https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/

Page caching. Btrfs does not support page cache sharing. This means that n containers accessing the same file require n copies to be cached. As a result, the btrfs driver may not be the best choice for PaaS and other high density container use cases.

https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

Memory usage. devicemapper is not the most memory efficient Docker storage driver. Launching n copies of the same container loads n copies of its files into memory. This can have a memory impact on your Docker host. As a result, the devicemapper storage driver may not be the best choice for PaaS and other high density use cases.

Perhaps it will be much more understandable if there is a table about this in https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

[thaJeztah]

Perhaps it will be much more understandable if there is a table about this in
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

Yes, I was thinking something like that; could you open an issue for that? (perhaps with a proposal as to what data you'd like to see in that table)?

[jokeyrhyme]

Just note, the original question for this Issue was whether memory (e.g. RAM and swap) was deduplicated or not. For example, if I run the same container twice, do I end up with 2 copies of glibc in memory?

It is good to share experiments for persistent storage and different filesystems, however, cheers.

[AkihiroSuda]

@thaJeztah
Oh, I found that https://github.com/docker/docker/blob/master/docs/userguide/storagedriver/images/driver-pros-cons.png was added 4 days ago (#19240), and it seems enough.

[AkihiroSuda]

@jokeyrhyme
Thank you for pointing.

Just note, the original question for this Issue was whether memory (e.g. RAM and swap) was deduplicated or not.

The answer for this original question is "It depends on the filesystem" for identical files on the same image, "No" for other things.
(Although I did not test for Docker, it can be "Yes" for almost all things (excluding static-linked apps) when you are using ksm_preload)

For example, if I run the same container twice, do I end up with 2 copies of glibc in memory?

So, the answer is "Yes" if you are using BtrFS or devicemapper, "No" if you are using AUFS or OverlayFS (or perhaps ZFS)

[grinapo]

@AkihiroSuda I fail to see why you keep talking about filesystems; this issue is about memory (KSM).

@ others: you may consider KSM not having suported is maybe not that bad: http://arstechnica.com/security/2016/08/new-attack-steals-private-crypto-keys-by-corrupting-data-in-computer-memory/ (updated rowhammer attack on KSM VMs)

[AkihiroSuda]

@grinapo
Actually the memory consumption (about files) depends on the filesystem you use.
https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/

Page Caching. OverlayFS supports page cache sharing. This means multiple containers accessing the same file can share a single page cache entry (or entries). This makes the overlay/overlay2 drivers efficient with memory and a good option for PaaS and other high density use cases.

https://docs.docker.com/engine/userguide/storagedriver/btrfs-driver/

Page caching. Btrfs does not support page cache sharing. This means that n containers accessing the same file require n copies to be cached. As a result, the btrfs driver may not be the best choice for PaaS and other high density container use cases.

[oveits]

@grinapo: I think you are right in pointing out that the original question was about shared memory pages not only for the RW layer of the UNION file system, but also for the real RAM. I just have come across the following links:

1. Virtuozzo describing how shared memory pages with copy-on-write works.
2. Memory Page sharing capability comparison between different virtualization technologies: interestingly, OpenVZ, Virtuozzo as well as LXC seem to support memory page sharing

I am not sure, whether LCD/LXC and/or Docker/LXC make use of this feature, though...

[jgstew]

If you have a docker host running on top of a hypervisor, then it should be possible for either the docker host or the hypervisor to do memory deduplication and compression.

• https://virtualizationreview.com/articles/2016/11/28/overcoming-the-ram-bottleneck.aspx
• https://en.wikipedia.org/wiki/Zram

[fabio-barile]

Hi
I am using ubuntu 16.04 with the last version of docker . My storage driver is aufs.
I am running around 30 containers on my laptop but the stats in
/sys/kernel/mm/ksm are always 0 excepted the run
It seems that ksm is not working well .
Can some of you share your own experience with Linux and ksm ?
Thanks

[tanjianfeng]

ksm needs explicit syscall, madvise(..., MADV_MERGEABLE) to work; so ksm can not work here I suppose. But it could be interesting if someone figures out a mechanism to enable this.

[jgstew]

That is what I was hoping for, a way to force it to assume madvise(..., MADV_MERGEABLE) for docker containers.

[srd424]

Note that KSM is only for private pages anyway, not page cache pages.

[devZer0]

ksm needs explicit syscall, madvise(..., MADV_MERGEABLE) to work; so ksm can not work here I suppose. But it could be interesting if someone figures out a mechanism to enable this.

not true anymore, as there is prctl() now and also support for KSM in systemd https://lwn.net/Articles/953141/

"That is where the prctl() flag, which was added for 6.4, came from. At the same time, systemd was modified to add the MemoryKSM parameter to enable KSM for a systemd service. The advantage of this approach is that the application code does not need to change at all to take advantage of KSM."

does that perhaps mean we can now enable KSM by simply adding MemoryKSM param to container/docker systemd service configuration file ?