"failed to write file content: out of memory" when trying new tar file snapshot feature

[dylanlanigansmith]

Description

I know this is feature is really only a day old, super excited about it, but was unable to reproduce! Hopefully this is helpful to diagnose either an issue in the implementation or documentation.

Expected:
Could use the new feature described here to run a container using an exported tarball as the upper-layer.

ie. runsc tar rootfs-upper --file snapshot.tar snapshot_me
then adding an annotation to spec config.json as described,
and then runsc run snapshotted is expected to work as described

Observed:

Container fails to start, exiting with an out_of_memory error

running container: starting container: starting root container: starting sandbox: failed to setupFS: creating mount namespace: mounting root with overlay: failed to create upper layer for overlay, opts: {Flags:{NoExec:false NoATime:false NoDev:false NoSUID:false} ReadOnly:false GetFilesystemOptions:{InternalMount:true Data: InternalData:{RootFileType:16384 RootSymlinkTarget: FilesystemType:<nil> Usage:<nil> MaxFilenameLen:0 MemoryFile:unwasteSmall:
-   0: [0x1000, 0xffffffffffffffff) => {}
unfreeSmall:
-   0: [0x0, 0x1000) => {0}
-   1: [0x40000000, 0xffffffffffffffff) => {0}
memAcct:
-   0: [0x0, 0x1000) => {0 0 false true 0}
 DisableDefaultSizeLimit:true AllowXattrPrefix:[] SourceTarFile:0xc00058c1d0}} Locked:false}: failed to make file ./test123: failed to write file content for ./test123: failed to write file content: out of memory

This happens in every combo I have tried, for example: with or without cgroups (and mem limits in turn), in every network mode, and even with default runsc spec. See below for steps to repro.

Steps to reproduce

I discovered this trying to use it with my own stuff, but for this report I used rootfs a generated in the same fashion as CNI Example for consistency.

1. Make rootfs (in same style as link above)

sudo mkdir -p bundle                                                                                                                                                                                                                             
cd bundle
sudo mkdir rootfs
sudo docker export $(docker create python) | sudo tar --same-owner -pxf - -C rootfs

2. Make initial spec config.json

runsc spec -- /bin/bash
# additions:
# edited readonly to false in root section
# added terminal: true in process section for convenience
# full spec can be seen in attached logs 

3. Run the container. Make some test files for the snapshot

- make a variety of files to include in snapshot
$ sudo runsc run snapshot_me                                                                                                                                                                                                                
root@runsc:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@runsc:/# mkdir -p snapshot/folder/to/include
root@runsc:/# echo "hello world" > !$/file.txt
echo "hello world" > snapshot/folder/to/include/file.txt
root@runsc:/# touch test123 FILE_IN_SNAPSHOT
root@runsc:/# cp /bin/sh /bin/mybettersh

4. From another terminal, create the snapshot tar as described in the snapshot example

$ sudo runsc tar rootfs-upper --file snapshot.tar snapshot_me                          
Serializing rootfs upper layer into a tar archive for container: snapshot_me, sandbox: snapshot

5. Again, following the example, add annotation with snapshot path to config.json to use the snapshot tar as upper layer

 "annotations": {
           "dev.gvisor.tar.rootfs.upper": "/home/dylan/sandbox/tmp/bundle/snapshot.tar"   
    }

6. Run a new container with the modified config.json

$ sudo runsc run snapshotted     
running container: starting container: starting root container: starting sandbox: failed to setupFS: creating mount namespace: mounting root with overlay: failed to create upper layer for overlay, opts: {Flags:{NoExec:false NoATime:false NoDev:false NoSUID:false} ReadOnly:false GetFilesystemOptions:{InternalMount:true Data: InternalData:{RootFileType:16384 RootSymlinkTarget: FilesystemType:<nil> Usage:<nil> MaxFilenameLen:0 MemoryFile:unwasteSmall:
-   0: [0x1000, 0xffffffffffffffff) => {}
unfreeSmall:
-   0: [0x0, 0x1000) => {0}
-   1: [0x40000000, 0xffffffffffffffff) => {0}
memAcct:
-   0: [0x0, 0x1000) => {0 0 false true 0}
 DisableDefaultSizeLimit:true AllowXattrPrefix:[] SourceTarFile:0xc00058c1d0}} Locked:false}: failed to make file ./test123: failed to write file content for ./test123: failed to write file content: out of memory

# additionally, demonstrate that host system has myriad memory available
$ free -h                                                                                  
               total        used        free      shared  buff/cache   available
Mem:            93Gi       5.0Gi        65Gi       258Mi        24Gi        88Gi
Swap:             0B          0B          0B

snapshot.log

runsc version

runsc version release-20251013.0-4-ga46c37df6e40
spec: 1.1.0-rc.1

docker version (if using docker)

N/A

uname

Linux 6.17.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 12 Oct 2025 12:45:18 +0000 x86_64 GNU/Linux

kubectl (if using Kubernetes)

N/A

repo state (if built from source)

release-20251013.0-4-ga46c37df6

runsc debug logs (if available)

[see attached file]

[dylanlanigansmith]

$ tar -tvf snapshot.tar                                                             
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./snapshot/
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./snapshot/folder/
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./snapshot/folder/to/
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./snapshot/folder/to/include/
-rw-r--r-- 0/0              12 2025-10-17 02:07 ./snapshot/folder/to/include/file.txt
-rw-r--r-- 0/0               0 2025-10-17 02:07 ./test123
-rw-r--r-- 0/0               0 2025-10-17 02:07 ./FILE_IN_SNAPSHOT
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./usr/
drwxr-xr-x 0/0               0 2025-10-17 02:07 ./usr/bin/
-rwxr-xr-x 0/0          129736 2025-10-17 02:07 ./usr/bin/mybettersh

meant to add this too! shows that tar seems fine.

[milantracy]

the feature a730d6c was merged on Oct 15 , it is not live in the version release-20251013.0-4-ga46c37df6e40, which means that the annotation you added should have no impact at all

in terms of OOM, it will be helpful if you share your config.json

[dylanlanigansmith]

I cloned master this evening and built it. Can confirm my binary has strings that match the commit. I also switched from self built to prebuilt from HEAD as described here, after submitting this, and can confirm still have the same issue. config json is default runsc spec with the two changes (read only: false, terminal: true) listed in steps to reproduce. You can also see it in attached log.

Here is confirmation I am using a build with this feature.

# binary from HEAD, which also matched the one I built today.

$ strings $(which runsc) | grep "dev.gvisor.tar.rootfs.upper"                          
...<too large, truncated>invalid file access type %qInvalid file access type %denable TX checksum offload.enable RX checksum offload.mount option skipped %q: %vApplying config bundles: %vdev.gvisor.tar.rootfs.upper

# Versus, 1013 binary for example:
$ wget https://storage.googleapis.com/gvisor/releases/release/latest/x86_64/runsc -O /tmp/runsc-latest
$ strings /tmp/runsc-latest | grep "dev.gvisor.tar.rootfs.upper"                            
[1] 

---

Here is a new error I was able to cause by messing with it some more, this seems like a good hint.. -overlay2=root:memory,size=1024m changes from OOM error to no space left on device... the size option/switch to memory being the thing that seems to make the difference.

$ runsc -debug -debug-to-user-log -debug-log log.log -network none -overlay2=root:memory,size=1024m run snapshot_me  
      
running container: starting container: starting root container: starting sandbox: failed to setupFS: creating mount namespace: mounting root with overlay: failed to create upper layer for overlay, opts: {Flags:{NoExec:false NoATime:false NoDev:false NoSUID:false} ReadOnly:false GetFilesystemOptions:{InternalMount:true Data:size=1024m InternalData:{RootFileType:16384 RootSymlinkTarget: FilesystemType:<nil> Usage:<nil> MaxFilenameLen:0 MemoryFile:<nil> DisableDefaultSizeLimit:true AllowXattrPrefix:[] SourceTarFile:0xc00058e1d0}} Locked:false}: 
failed to make file ./TEST failed to write file content for ./TEST: failed to write file content: no space left on device

[milantracy]

i think i found the problem,

when untaring, it doens't handle the empty files properly, which is ./test123 and ./FILE_IN_SNAPSHOT in your tar file.

i will send a fix soon, btw, let me how it works for your when you remove those empty files

[dylanlanigansmith]

yeah that is it. empty files breaks it.

$ sudo runsc run snapshot_me   
root@runsc:/# mkdir -p NO/EMPTY/FILES
root@runsc:/# echo "helllo" > !$/hello
echo "helllo" > NO/EMPTY/FILES/hello
root@runsc:/# echo "not empty" > FILE1
root@runsc:/# echo "not empty" > FILE2
root@runsc:/# echo "not empty" > FILE3
root@runsc:/# echo "not empty" > FILE4
root@runsc:/# echo "not empty" > FILE5
root@runsc:/# cp /bin/sh /bin/mysh

$ sudo runsc tar rootfs-upper --file /tmp/snapshot.tar snapshot_me 
$ sudo runsc -debug -debug-to-user-log -debug-log log.log run snapshotted                                                 
root@runsc:/# ls
FILE1  FILE2  FILE3  FILE4  FILE5  NO  bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run	sbin  srv  sys	tmp  usr  var
root@runsc:/# 

[dylanlanigansmith]

I fixed it. Made a tiny PR. Thanks @milantracy