Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dump huge tmpfs ghost files as shmem segments #230

Closed
hodiapa opened this issue Oct 17, 2016 · 12 comments

Comments

Projects
None yet
4 participants
@hodiapa
Copy link

commented Oct 17, 2016

I am referring to this issue: moby/moby#27200
I did following:

vagrant@vagrant:~$ docker run -d --security-opt seccomp:unconfined --name nas1 spirals/npb3.3.1 -np 16 ep.C.16 
4b32577ba2ed2a583ced5482ef50cc63b3cb0b4067dad46385e5eb2de2ea5b7f

Error:

vagrant@vagrant:~$ docker checkpoint create --leave-running nas1 chk2
Error response from daemon: Cannot checkpoint container nas1: rpc error: code = 2 desc = exit status 1: "criu failed: type NOTIFY errno 0\nlog file: /var/lib/docker/containers/4b32577ba2ed2a583ced5482ef50cc63b3cb0b4067dad46385e5eb2de2ea5b7f/checkpoints/chk2/criu.work/dump.log\n"

dump: https://gist.github.com/hodiapa/2c46ba21f35153fbb2bc2f6804f7363b

docker info:

vagrant@vagrant:~$ docker info
Containers: 32
 Running: 2
 Paused: 0
 Stopped: 30
Images: 70
Server Version: 1.13.0-dev
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 227
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-31-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: vagrant
ID: PIGT:R37S:A7DH:VSV5:NA4O:5U7G:YJA6:VUYQ:BECX:ULZC:PKAD:TFH7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

I was told @boucher that the lines of interest in the criu dump are, and this issue was not specific to docker, but rather CRIU:

(00.480204) Error (files-reg.c:683): Can't dump ghost file /dev/shm/mpich_shar_tmpGj4i0H of 82859216 size, increase limit
(00.480209) Error (cr-dump.c:1365): Dump mappings (pid: 28053) failed with -1
@xemul

This comment has been minimized.

Copy link
Member

commented Oct 19, 2016

Damn :( We have 82Mb ghost file in Docker root. And it sits in shmem...
You can ... work-around this with --ghost-limit 100000000, but this will make 82Megs file be copied into your images.

Proper fix would be to check this file for holes and only save data as it's effectively a shmem.

@hodiapa

This comment has been minimized.

Copy link
Author

commented Oct 19, 2016

Thanks @xemul for pointing that out. But where do I set this in the

docker checkpoint create --leave-running nas1 chk2

command?

@xemul

This comment has been minimized.

Copy link
Member

commented Oct 19, 2016

That's probably a question to @boucher or @avagin :) Gents, what's the way to pass custom criu options via Docker CLI?

@boucher

This comment has been minimized.

Copy link

commented Oct 19, 2016

There's no current way to specify custom options, and we don't really have
any specific plans for this right now. I'm open to suggestions, though
ultimately the RunC/Docker team will need to be convinced.

On Wed, Oct 19, 2016 at 2:48 PM Pavel Emelyanov notifications@github.com
wrote:

That's probably a question to @boucher https://github.com/boucher or
@avagin https://github.com/avagin :) Gents, what's the way to pass
custom criu options via Docker CLI?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xemul/criu/issues/230#issuecomment-254950878, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABWMb4aBnNan2t3QtMVSdqqGW1LskV2ks5q1pAigaJpZM4KZEZ7
.

@xemul

This comment has been minimized.

Copy link
Member

commented Oct 19, 2016

Heh :) It looks like Adrian's suggestion to have /etc/criu.conf where custom options can be stored would be the only way to go :D

However, for ghost size limit we can add an environment variable :\ @hodiapa that'd be the fastest way to move forward. If you could cook the patch it'd speed things up significantly.

@hodiapa

This comment has been minimized.

Copy link
Author

commented Oct 19, 2016

I definitely see a need for passing custom options to criu, somehow to resolve this issue :) . What other options are there? maybe the CRIU library could reads these options from some custom configurations files?

@xemul

This comment has been minimized.

Copy link
Member

commented Oct 19, 2016

@hodiapa yes, /etc/criu.conf is something that was suggested on the mailing list (https://lists.openvz.org/pipermail/criu/2016-August/030819.html), so if you could join the effort, that'd be awesome :)

@hodiapa

This comment has been minimized.

Copy link
Author

commented Oct 20, 2016

@boucher : Is this the location in the sourcecode to propose adding additional custom options in the runC, if we need to include in the RunC? https://github.com/opencontainers/runc/blob/52454cf90807727923b375afd7d84ac8fd523959/libcontainer/criu_opts_unix.go#L25

@boucher

This comment has been minimized.

Copy link

commented Oct 25, 2016

Truthfully, it's rather complex. You need to create command line runc
options that translate into criu swrk / protobuff options. Then you need to
pass those options all the way up the chain to docker (through containerd).

On Thu, Oct 20, 2016 at 9:57 AM hodiapa notifications@github.com wrote:

@boucher https://github.com/boucher : Is this the location in the
sourcecode to specify custom options in the runC?
https://github.com/opencontainers/runc/blob/52454cf90807727923b375afd7d84ac8fd523959/libcontainer/criu_opts_unix.go#L25


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/xemul/criu/issues/230#issuecomment-255164610, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABWMZ59P6XfkUqbMynzzRiCDWKLaMpBks5q151mgaJpZM4KZEZ7
.

@xemul

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

Cc @avagin . This tmpfs file should be dumped as regular shmem segment.

@xemul xemul changed the title can't checkpoint a running docker container dump huge tmpfs ghost files are shmem segments Mar 3, 2017

@xemul xemul changed the title dump huge tmpfs ghost files are shmem segments dump huge tmpfs ghost files as shmem segments Mar 28, 2017

@xemul xemul removed the docker label Mar 28, 2017

criupatchwork pushed a commit to criupatchwork/criu that referenced this issue Jun 19, 2017

ghost: Add chunked ghost files
If the ghost file is too big, it might make sence to try seeking
for holes in it, thus reducing the image size.

We've seen this once for tmpfs files in issue checkpoint-restore#230.

Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

avagin added a commit that referenced this issue Jun 21, 2017

ghost: Add chunked ghost files
If the ghost file is too big, it might make sence to try seeking
for holes in it, thus reducing the image size.

We've seen this once for tmpfs files in issue #230.

Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
@xemul

This comment has been minimized.

Copy link
Member

commented Jun 22, 2017

Will be in 3.3

@xemul xemul closed this Jun 22, 2017

avagin added a commit to avagin/criu that referenced this issue Jul 18, 2017

ghost: Add chunked ghost files
If the ghost file is too big, it might make sence to try seeking
for holes in it, thus reducing the image size.

We've seen this once for tmpfs files in issue checkpoint-restore#230.

Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

avagin added a commit to avagin/criu that referenced this issue Jul 18, 2017

ghost: Add chunked ghost files
If the ghost file is too big, it might make sence to try seeking
for holes in it, thus reducing the image size.

We've seen this once for tmpfs files in issue checkpoint-restore#230.

Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>
Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
@farooqte96

This comment has been minimized.

Copy link

commented May 23, 2018

@xemul Can you tell how to set this ghost-limit option with docker as i am getting same issue when checkpointed image size is large. I am using docker version 17.06 ee with criu version 3.8.1. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.