Skip to content

mv prompts about overwriting non-existent files #11794

New issue
New issue
Closed
#11798
Closed
mv prompts about overwriting non-existent files#11794
#11798
Labels
type: bugSomething isn't working
@dseomn

Description

@dseomn
dseomn
opened on Jun 5, 2025

Description

Sometimes the mv command prompts about overwriting a non-existent file:

* dseomn@8435c984468e:/tmp/tmp.xPFwwVcgKD$ ls -laR
.:
total 0
drwx------ 2 dseomn dseomn   100 Jun  5 20:34 .
drwxrwxrwt 2 root   root   72260 Jun  5 20:33 ..
drwxr-xr-x 2 dseomn dseomn    60 Jun  5 20:34 foo

./foo:
total 0
drwxr-xr-x 2 dseomn dseomn  60 Jun  5 20:34 .
drwx------ 2 dseomn dseomn 100 Jun  5 20:34 ..
-rw-r--r-- 1 dseomn dseomn   0 Jun  5 20:34 bar
* dseomn@8435c984468e:/tmp/tmp.xPFwwVcgKD$ mv foo/bar quux
mv: replace 'quux', overriding mode 5000 (--S-----T)?

The actual mode it shows is different each time, and occasionally it works, which makes me think it might be reading the mode from uninitialized memory. Here's part of the strace log:

newfstatat(AT_FDCWD, "quux", 0x7ee458b79eb0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
geteuid()                               = 1000
faccessat2(AT_FDCWD, "quux", W_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
faccessat2(AT_FDCWD, "quux", W_OK, AT_EACCESS) = -1 ENOENT (No such file or directory)
write(2, "mv: replace 'quux', overriding m"..., 54mv: replace 'quux', overriding mode 0000 (---------)? ) = 54

Steps to reproduce

I created the files and directories above using mktemp -d, mkdir foo, and touch foo/bar. runsc is running rootless with -ignore-cgroups and -network=host.

runsc version

runsc version 0.0~20240729.0
spec: 1.2.0

docker version (if using docker)

Client:        Podman Engine
Version:       5.4.2
API Version:   5.4.2
Go Version:    go1.24.2
Built:         Sat May 24 14:25:04 2025
Build Origin:  Debian
OS/Arch:       linux/amd64

uname

Linux solaria 6.12.27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.27-1 (2025-05-06) x86_64 GNU/Linux

kubectl (if using Kubernetes)

N/A

repo state (if built from source)

N/A

runsc debug logs (if available)


    
      
      
    
  
Activity
dseomn
added 
type: bugSomething isn't working
 on Jun 5, 2025
ayushr2
ayushr2 commented on Jun 5, 2025 
@ayushr2
ayushr2
on Jun 5, 2025
Collaborator
Can you provide the full debug logs?
dseomn
dseomn commented on Jun 5, 2025 
@dseomn
dseomn
on Jun 5, 2025
Author
When I restarted the container (with the debug logging options), it didn't happen right away. However, I tried to recreate what I had done before, and got the error.


Here's the relevant part of a shell script that I was running. The m17n/m17n-lib directory is a git clone of https://cgit.git.savannah.gnu.org/cgit/m17n/m17n-lib.git/tree/src/ with some local modifications, mounted with --volume="$HOME/Code/m17n:$HOME/m17n:ro".


podman exec --tty m17n bash -l -c '
  set -euxo pipefail
  rsync -va --del m17n/ work/
  cd work/m17n-lib
  ./bootstrap.sh
  ./configure
  make --jobs="$(nproc)" --quiet  # TODO
  # make
'


    
      
      
    
  


Before this issue happened the first time (I think), I had started to run that, then realized that I was missing something I wanted to build, so I pressed Ctrl-C to try to cancel it. That didn't work (I think because I didn't pass --interactive to podman), so I killed the podman process instead. After killing it and running it again, I started to get the bug. Here's the end of the output from running it again after killing it, which I think is the same issue with mv getting the wrong stat results:


mv: cannot overwrite directory 'Makefile' with non-directory './confpNR0ei/out'
config.status: error: could not create Makefile


    
      
      
    
  


I got an error uploading the logs to GitHub, maybe because of the file size, so here's a Google Drive link: https://drive.google.com/file/d/1hMZIoSJpgQAwvkA9Mo7ejVv1dze8lgOY/view?usp=sharing
dseomn
dseomn commented on Jun 5, 2025 
@dseomn
dseomn
on Jun 5, 2025
Author
It just happened again after restarting the container, without killing any podman processes this time.


mv: cannot overwrite directory 'example/Makefile' with non-directory './confe5MKoz/out'
config.status: error: could not create example/Makefile
dseomn@solaria:~ (dfgr: personal *%=) [1]$ podman exec -ti m17n bash -l
* dseomn@e0a6108b2444:~$ ls -l work/m17n-lib/example/
total 248
-rw-rw-r-- 1 dseomn dseomn 23157 May 12 03:26 ChangeLog
-rw-rw-r-- 1 dseomn dseomn  1521 May 12 03:26 HELLO-ja.utf8
-rw-rw-r-- 1 dseomn dseomn  3511 May 12 03:26 HELLO-ja.xml
-rw-rw-r-- 1 dseomn dseomn  4187 May 12 03:26 HELLO.html
-rw-rw-r-- 1 dseomn dseomn  2076 May 12 03:26 HELLO.utf8
-rw-rw-r-- 1 dseomn dseomn  4807 May 12 03:26 HELLO.xml
-rw-rw-r-- 1 dseomn dseomn  4936 May 12 03:26 M17NEdit.ja
-rw-rw-r-- 1 dseomn dseomn  3526 Jun  5 20:01 Makefile.am
-rw-r--r-- 1 dseomn dseomn 35633 Jun  5 22:25 Makefile.in
-rw-rw-r-- 1 dseomn dseomn 11059 May 12 03:26 mconv.c
-rw-rw-r-- 1 dseomn dseomn  6133 May 12 03:26 mdate.c
-rw-rw-r-- 1 dseomn dseomn 22030 May 12 03:26 mdump.c
-rw-rw-r-- 1 dseomn dseomn 86617 May 12 03:26 medit.c
-rw-rw-r-- 1 dseomn dseomn 10177 May 12 03:26 mimx-anthy.c
-rw-rw-r-- 1 dseomn dseomn  6520 May 12 03:26 mimx-ispell.c
-rw-rw-r-- 1 dseomn dseomn 11827 Jun  5 22:25 minputtest.c
-rw-rw-r-- 1 dseomn dseomn 12256 May 12 03:26 mview.c


    
      
      
    
  
ayushr2
ayushr2 commented on Jun 5, 2025 
@ayushr2
ayushr2
on Jun 5, 2025
Collaborator
I0605 17:38:53.560142  813506 strace.go:573] [ 17950: 17950] mv E renameat2(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e79 ./confpNR0ei/out, 0xffffff9c, 0x7ed005894e8a Makefile, 0x1)
I0605 17:38:53.560156  813506 strace.go:611] [ 17950: 17950] mv X renameat2(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e79 ./confpNR0ei/out, 0xffffff9c, 0x7ed005894e8a Makefile, 0x1) = 0 (0x0) errno=17 (file exists) (10.479µs)
I0605 17:38:53.560178  813506 strace.go:570] [ 17950: 17950] mv E openat(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e8a Makefile, O_RDONLY|O_DIRECTORY|O_PATH, 0o0)
I0605 17:38:53.560183  813506 strace.go:608] [ 17950: 17950] mv X openat(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e8a Makefile, O_RDONLY|O_DIRECTORY|O_PATH, 0o0) = 0 (0x0) errno=2 (no such file or directory) (1.89µs)
D0605 17:38:53.560190  813506 usertrap_amd64.go:210] [ 17950: 17950] Found the pattern at ip 7ec8ad4251c3:sysno 262
D0605 17:38:53.560192  813506 usertrap_amd64.go:122] [ 17950: 17950] Allocate a new trap: 0xc0080b6d80 19
D0605 17:38:53.560195  813506 usertrap_amd64.go:223] [ 17950: 17950] Apply the binary patch addr 7ec8ad4251c3 trap addr 615f0 ([184 6 1 0 0 15 5] -> [255 36 37 240 21 6 0])
I0605 17:38:53.560200  813506 strace.go:570] [ 17950: 17950] mv E newfstatat(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e79 ./confpNR0ei/out, 0x7ed0058946c0, 0x100)
I0605 17:38:53.560210  813506 strace.go:608] [ 17950: 17950] mv X newfstatat(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e79 ./confpNR0ei/out, 0x7ed0058946c0 {dev=17, ino=20207, mode=S_IFREG|0o644, nlink=1, uid=1000, gid=1000, rdev=0, size=35459, blksize=4096, blocks=70, atime=2025-06-05 17:38:53.533903641 -0400 EDT, mtime=2025-06-05 17:38:53.548285551 -0400 EDT, ctime=2025-06-05 17:38:53.548285551 -0400 EDT}, 0x100) = 0 (0x0) (4.02µs)
I0605 17:38:53.560216  813506 strace.go:570] [ 17950: 17950] mv E newfstatat(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e8a Makefile, 0x7ed005894750, 0x100)
I0605 17:38:53.560220  813506 strace.go:608] [ 17950: 17950] mv X newfstatat(AT_FDCWD /home/dseomn/work/m17n-lib, 0x7ed005894e8a Makefile, 0x7ed005894750, 0x100) = 0 (0x0) errno=2 (no such file or directory) (1.164µs)
D0605 17:38:53.560230  813506 usertrap_amd64.go:210] [ 17950: 17950] Found the pattern at ip 7ec8ad429a59:sysno 72
D0605 17:38:53.560233  813506 usertrap_amd64.go:122] [ 17950: 17950] Allocate a new trap: 0xc0080b6d80 20
D0605 17:38:53.560236  813506 usertrap_amd64.go:223] [ 17950: 17950] Apply the binary patch addr 7ec8ad429a59 trap addr 61640 ([184 72 0 0 0 15 5] -> [255 36 37 64 22 6 0])
I0605 17:38:53.560241  813506 strace.go:567] [ 17950: 17950] mv E fcntl(0x1 host:[12], 0x3, 0x55e83daf1c50)
I0605 17:38:53.560245  813506 strace.go:605] [ 17950: 17950] mv X fcntl(0x1 host:[12], 0x3, 0x55e83daf1c50) = 2 (0x2) (879ns)
I0605 17:38:53.560254  813506 strace.go:567] [ 17950: 17950] mv E write(0x2 host:[12], 0x7ed0058942a0 "mv: ", 0x4)
I0605 17:38:53.560265  813506 strace.go:605] [ 17950: 17950] mv X write(0x2 host:[12], ..., 0x4) = 4 (0x4) (8.168µs)
I0605 17:38:53.560272  813506 strace.go:567] [ 17950: 17950] mv E write(0x2 host:[12], 0x7ed005894390 "cannot overwrite directory 'Makefile' with non-directory './confpNR0ei/out'", 0x4b)
I0605 17:38:53.560276  813506 strace.go:605] [ 17950: 17950] mv X write(0x2 host:[12], ..., 0x4b) = 75 (0x4b) (1.971µs)
I0605 17:38:53.560283  813506 strace.go:567] [ 17950: 17950] mv E write(0x2 host:[12], 0x7ec8ad50d563 "\n", 0x1)
I0605 17:38:53.560287  813506 strace.go:605] [ 17950: 17950] mv X write(0x2 host:[12], ..., 0x1) = 1 (0x1) (1.782µs)


    
      
      
    
  


IIUC, we are trying to move /home/dseomn/work/m17n-lib/confpNR0ei/out to /home/dseomn/work/m17n-lib/Makefile.

/home/dseomn/work/m17n-lib/confpNR0ei/out is a regular file that exists.renameat2 fails with EEXIST but openat() and newfstatat() confirms that /home/dseomn/work/m17n-lib/Makefile does not exist.


I can see that /home/dseomn/work/m17n-lib/Makefile existed in the past:


I0605 17:37:58.024630  813506 strace.go:570] [ 8559: 8559] rsync E newfstatat(AT_FDCWD /home/dseomn/work, 0x7ed70bfeb6c0 m17n-lib/Makefile, 0x7ed70bfeb630, 0x100)
I0605 17:37:58.024638  813506 strace.go:608] [ 8559: 8559] rsync X newfstatat(AT_FDCWD /home/dseomn/work, 0x7ed70bfeb6c0 m17n-lib/Makefile, 0x7ed70bfeb630 {dev=17, ino=14250, mode=S_IFREG|0o644, nlink=1, uid=1000, gid=1000, rdev=0, size=35459, blksize=4096, blocks=70, atime=2025-06-05 17:36:40.440785236 -0400 EDT, mtime=2025-06-05 17:36:38.64897331 -0400 EDT, ctime=2025-06-05 17:36:38.661750223 -0400 EDT}, 0x100) = 0 (0x0) (2.451µs)


    
      
      
    
  


But it was soon deleted:


I0605 17:37:58.028951  813506 strace.go:561] [ 8559: 8559] rsync E unlink(0x7ed70bff0960 m17n-lib/Makefile)
I0605 17:37:58.028958  813506 strace.go:599] [ 8559: 8559] rsync X unlink(0x7ed70bff0960 m17n-lib/Makefile) = 0 (0x0) (4.273µs)


    
      
      
    
  


It is confusing with renameat2() would fail with EEXIST in that case.
dseomn
dseomn commented on Jun 6, 2025 
@dseomn
dseomn
on Jun 6, 2025
Author


I can see that /home/dseomn/work/m17n-lib/Makefile existed in the past:

…

But it was soon deleted:




That sounds right. It should be generated during the configure/build process, and I'm deleting all generated files and rebuilding because it's fast enough and I don't want to deal with stale build artifacts.
ayushr2
ayushr2 commented on Jun 6, 2025 
@ayushr2
ayushr2
on Jun 6, 2025
Collaborator
@dseomn Could you:


    

  • Provide a full reproducer? I know you shared the shell script from above, but that seems to be exec-ing into an already running container. Could you provide your setup? Like what image you are using?
    • 

  • In your first comment, you shared a reproducer with mktemp -d, mkdir foo, touch foo/bar, ls -laR and mv foo/bar quux. I tried this multiple times but can't repro. You mentioned that this happens frequently enough. The reproducer I am trying is: docker run --runtime=op --rm -it ubuntu bash -c 'cd "$(mktemp -d)" && pwd && mkdir foo && touch foo/bar && ls -laR && mv foo/bar quux'.
    • 

  • I see that you are using the 20240729 release. Could you try with the latest runsc release and see if it still repros?
    • 

  • Could you set runsc flag --overlay2=none and check? This is just to narrow down the issue to overlayfs/tmpfs.
    • 

dseomn
dseomn commented on Jun 6, 2025 
@dseomn
dseomn
on Jun 6, 2025
Author


Provide a full reproducer? I know you shared the shell script from above, but that seems to be exec-ing into an already running container. Could you provide your setup? Like what image you are using?






In your first comment, you shared a reproducer with mktemp -d, mkdir foo, touch foo/bar, ls -laR and mv foo/bar quux. I tried this multiple times but can't repro. You mentioned that this happens frequently enough. The reproducer I am trying is: docker run --runtime=op --rm -it ubuntu bash -c 'cd "$(mktemp -d)" && pwd && mkdir foo && touch foo/bar && ls -laR && mv foo/bar quux'.




Here's a one-liner that reliably reproduces the bug for me in some containers:


c=gvisor-bug; podman exec $c rm -rf /tmp/bug-test && podman exec $c mkdir /tmp/bug-test && podman exec $c touch /tmp/bug-test/foo && for i in {1..50}; do podman exec $c mv /tmp/bug-test/{foo,bar}; podman exec $c mv /tmp/bug-test/{bar,foo}; done


    
      
      
    
  


In this container, I don't get the bug:


podman run -d --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 debian:testing sleep 3600


    
      
      
    
  


In this container, I do get the bug:


podman run -d --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 localhost/dev-environment:latest sleep 3600


    
      
      
    
  


localhost/dev-environment was built with https://github.com/dseomn/dotfiles/blob/7074061500b7b81bdadfa5f0e290ff72ec1e6664/.local/bin/dev-environment-build using https://github.com/dseomn/dotfiles/blob/7074061500b7b81bdadfa5f0e290ff72ec1e6664/.local/share/dotfiles/dev-environment/Containerfile


I added the --user, --workdir, and --env flags and specified a command to get rid of those differences between the two images. I don't think any of the other differences are relevant, except that they add additional layers and size to the image?


I tried the same command with new images generated from these to see if the additional layers mattered, but I didn't get the bug:


FROM debian:testing

RUN touch /file-to-make-another-layer

    
      
      
    
  


FROM debian:testing

RUN touch /file-to-make-another-layer-1
RUN touch /file-to-make-another-layer-2
RUN touch /file-to-make-another-layer-3
RUN touch /file-to-make-another-layer-4
RUN touch /file-to-make-another-layer-5
RUN touch /file-to-make-another-layer-6
RUN touch /file-to-make-another-layer-7
RUN touch /file-to-make-another-layer-8
RUN touch /file-to-make-another-layer-9
RUN touch /file-to-make-another-layer-10

    
      
      
    
  




I see that you are using the 20240729 release. Could you try with the latest runsc release and see if it still repros?




I want to work on other stuff today, but I can try that in a bit.




Could you set runsc flag --overlay2=none and check? This is just to narrow down the issue to overlayfs/tmpfs.




With this container, I don't see the bug:


podman run -d --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 --runtime-flag=overlay2=none localhost/dev-environment:latest sleep 3600


    
      
      
    
  


So that seems to be a good workaround for now, thank you!
ayushr2
ayushr2 commented on Jun 6, 2025 
@ayushr2
ayushr2
on Jun 6, 2025
Collaborator
Thanks @dseomn




I want to work on other stuff today, but I can try that in a bit.




I can reproduce this with the latest build. So no action needed from you. I will investigate this.




With this container, I don't see the bug:




Ah so this is indeed an overlayfs OR tmpfs bug.
ayushr2
ayushr2 commented on Jun 6, 2025 
@ayushr2
ayushr2
on Jun 6, 2025
Collaborator
I think I figured out the issue. This is a bug in our overlayfs implementation. This occurs when using renameat2(2) syscall with RENAME_NOREPLACE flag and trying to rename a file into a position which contains a whiteout. Overlayfs correctly identifies the whiteout and considers it an empty slot. But when it calls into the upper layer to actually do the rename work, it passes the same RENAME_NOREPLACE flag. tmpfs (upper layer) does not treat the whiteout (which is a character device) specially and errors out with EEXIST.


Usually this does not cause any issues because the mv binary just retries the rename without the RENAME_NOREPLACE flag and that syscall succeeds. This is evidenced in the logs here from the non-error-printing attempts:


I0606 22:22:51.959754       1 strace.go:573] [   7:   7] mv E renameat2(AT_FDCWD /, 0x7eb4ca1fff23 /tmp/bug-test/bar, 0xffffff9c, 0x7eb4ca1fff35 /tmp/bug-test/foo, 0x1)
I0606 22:22:51.959843       1 strace.go:611] [   7:   7] mv X renameat2(AT_FDCWD /, 0x7eb4ca1fff23 /tmp/bug-test/bar, 0xffffff9c, 0x7eb4ca1fff35 /tmp/bug-test/foo, 0x1) = 0 (0x0) errno=17 (file exists) (78µs)
I0606 22:22:51.959913       1 strace.go:570] [   7:   7] mv E openat(AT_FDCWD /, 0x7eb4ca1fff35 /tmp/bug-test/foo, O_RDONLY|O_DIRECTORY|O_PATH, 0o0)
I0606 22:22:51.959934       1 strace.go:608] [   7:   7] mv X openat(AT_FDCWD /, 0x7eb4ca1fff35 /tmp/bug-test/foo, O_RDONLY|O_DIRECTORY|O_PATH, 0o0) = 0 (0x0) errno=2 (no such file or directory) (10.88µs)
D0606 22:22:51.959952       1 usertrap_amd64.go:210] [   7:   7] Found the pattern at ip 7eaaf7a091c3:sysno 262
D0606 22:22:51.959964       1 usertrap_amd64.go:122] [   7:   7] Allocate a new trap: 0xc0005969c0 21
D0606 22:22:51.959971       1 usertrap_amd64.go:223] [   7:   7] Apply the binary patch addr 7eaaf7a091c3 trap addr 61690 ([184 6 1 0 0 15 5] -> [255 36 37 144 22 6 0])
I0606 22:22:51.959981       1 strace.go:570] [   7:   7] mv E newfstatat(AT_FDCWD /, 0x7eb4ca1fff23 /tmp/bug-test/bar, 0x7eb4ca1ff7a0, 0x100)
I0606 22:22:51.960008       1 strace.go:608] [   7:   7] mv X newfstatat(AT_FDCWD /, 0x7eb4ca1fff23 /tmp/bug-test/bar, 0x7eb4ca1ff7a0 {dev=17, ino=5, mode=S_IFREG|0o644, nlink=1, uid=65534, gid=65534, rdev=0, size=0, blksize=4096, blocks=0, atime=2025-06-06 22:22:50.511453906 +0000 UTC, mtime=2025-06-06 22:22:50.511453906 +0000 UTC, ctime=2025-06-06 22:22:51.939748053 +0000 UTC}, 0x100) = 0 (0x0) (9.4µs)
I0606 22:22:51.960031       1 strace.go:570] [   7:   7] mv E newfstatat(AT_FDCWD /, 0x7eb4ca1fff35 /tmp/bug-test/foo, 0x7eb4ca1ff830, 0x100) 
I0606 22:22:51.960063       1 strace.go:608] [   7:   7] mv X newfstatat(AT_FDCWD /, 0x7eb4ca1fff35 /tmp/bug-test/foo, 0x7eb4ca1ff830, 0x100) = 0 (0x0) errno=2 (no such file or directory) (6.6µs)
D0606 22:22:51.960097       1 usertrap_amd64.go:210] [   7:   7] Found the pattern at ip 7eaaf7968453:sysno 264
D0606 22:22:51.960109       1 usertrap_amd64.go:122] [   7:   7] Allocate a new trap: 0xc0005969c0 22
D0606 22:22:51.960120       1 usertrap_amd64.go:223] [   7:   7] Apply the binary patch addr 7eaaf7968453 trap addr 616e0 ([184 8 1 0 0 15 5] -> [255 36 37 224 22 6 0])
I0606 22:22:51.960139       1 strace.go:570] [   7:   7] mv E renameat(AT_FDCWD /, 0x7eb4ca1fff23 /tmp/bug-test/bar, 0xffffff9c, 0x7eb4ca1fff35 /tmp/bug-test/foo)
I0606 22:22:51.960238       1 strace.go:608] [   7:   7] mv X renameat(AT_FDCWD /, 0x7eb4ca1fff23 /tmp/bug-test/bar, 0xffffff9c, 0x7eb4ca1fff35 /tmp/bug-test/foo) = 0 (0x0) (85.19µs)


    
      
      
    
  


But sometimes, the mv binary does not do this retry step, as evidenced in these logs:


I0606 22:22:52.132177       1 strace.go:573] [  15:  15] mv E renameat2(AT_FDCWD /, 0x7edaf5e4cf23 /tmp/bug-test/bar, 0xffffff9c, 0x7edaf5e4cf35 /tmp/bug-test/foo, 0x1)
I0606 22:22:52.132266       1 strace.go:611] [  15:  15] mv X renameat2(AT_FDCWD /, 0x7edaf5e4cf23 /tmp/bug-test/bar, 0xffffff9c, 0x7edaf5e4cf35 /tmp/bug-test/foo, 0x1) = 0 (0x0) errno=17 (file exists) (79.39µs)
I0606 22:22:52.132360       1 strace.go:570] [  15:  15] mv E openat(AT_FDCWD /, 0x7edaf5e4cf35 /tmp/bug-test/foo, O_RDONLY|O_DIRECTORY|O_PATH, 0o0)
I0606 22:22:52.132378       1 strace.go:608] [  15:  15] mv X openat(AT_FDCWD /, 0x7edaf5e4cf35 /tmp/bug-test/foo, O_RDONLY|O_DIRECTORY|O_PATH, 0o0) = 0 (0x0) errno=2 (no such file or directory) (9.57µs)
D0606 22:22:52.132397       1 usertrap_amd64.go:210] [  15:  15] Found the pattern at ip 7eb73a7701c3:sysno 262
D0606 22:22:52.132404       1 usertrap_amd64.go:122] [  15:  15] Allocate a new trap: 0xc000a941e0 21
D0606 22:22:52.132410       1 usertrap_amd64.go:223] [  15:  15] Apply the binary patch addr 7eb73a7701c3 trap addr 68690 ([184 6 1 0 0 15 5] -> [255 36 37 144 134 6 0])
I0606 22:22:52.132420       1 strace.go:570] [  15:  15] mv E newfstatat(AT_FDCWD /, 0x7edaf5e4cf23 /tmp/bug-test/bar, 0x7edaf5e4c7a0, 0x100)
I0606 22:22:52.132453       1 strace.go:608] [  15:  15] mv X newfstatat(AT_FDCWD /, 0x7edaf5e4cf23 /tmp/bug-test/bar, 0x7edaf5e4c7a0 {dev=17, ino=5, mode=S_IFREG|0o644, nlink=1, uid=65534, gid=65534, rdev=0, size=0, blksize=4096, blocks=0, atime=2025-06-06 22:22:50.511453906 +0000 UTC, mtime=2025-06-06 22:22:50.511453906 +0000 UTC, ctime=2025-06-06 22:22:52.11114143 +0000 UTC}, 0x100) = 0 (0x0) (14.89µs)
I0606 22:22:52.132488       1 strace.go:570] [  15:  15] mv E newfstatat(AT_FDCWD /, 0x7edaf5e4cf35 /tmp/bug-test/foo, 0x7edaf5e4c830, 0x100)
I0606 22:22:52.132504       1 strace.go:608] [  15:  15] mv X newfstatat(AT_FDCWD /, 0x7edaf5e4cf35 /tmp/bug-test/foo, 0x7edaf5e4c830, 0x100) = 0 (0x0) errno=2 (no such file or directory) (9.74µs)
D0606 22:22:52.132521       1 usertrap_amd64.go:210] [  15:  15] Found the pattern at ip 7eb73a774a59:sysno 72
D0606 22:22:52.132530       1 usertrap_amd64.go:122] [  15:  15] Allocate a new trap: 0xc000a941e0 22
D0606 22:22:52.132537       1 usertrap_amd64.go:223] [  15:  15] Apply the binary patch addr 7eb73a774a59 trap addr 686e0 ([184 72 0 0 0 15 5] -> [255 36 37 224 134 6 0])
I0606 22:22:52.132546       1 strace.go:567] [  15:  15] mv E fcntl(0x1 host:[14], 0x3, 0x56484d3c1c50)
I0606 22:22:52.132552       1 strace.go:605] [  15:  15] mv X fcntl(0x1 host:[14], 0x3, 0x56484d3c1c50) = 1 (0x1) (830ns)
I0606 22:22:52.132604       1 strace.go:567] [  15:  15] mv E write(0x2 host:[15], 0x7edaf5e4c380 "mv: ", 0x4)
I0606 22:22:52.132628       1 strace.go:605] [  15:  15] mv X write(0x2 host:[15], ..., 0x4) = 4 (0x4) (17.12µs)
I0606 22:22:52.132654       1 strace.go:567] [  15:  15] mv E write(0x2 host:[15], 0x7edaf5e4c470 "cannot overwrite directory '/tmp/bug-test/foo' with non-directory '/tmp/bug-test/bar'", 0x55)
I0606 22:22:52.132663       1 strace.go:605] [  15:  15] mv X write(0x2 host:[15], ..., 0x55) = 85 (0x55) (2.19µs)
I0606 22:22:52.132679       1 strace.go:567] [  15:  15] mv E write(0x2 host:[15], 0x7eb73a858563 "\n", 0x1)
I0606 22:22:52.132687       1 strace.go:605] [  15:  15] mv X write(0x2 host:[15], ..., 0x1) = 1 (0x1) (1.87µs)


    
      
      
    
  


I also can't reproduce these failures with images like debian:testing and ubuntu:latest (as you pointed out). This is only reproducible with the custom image from your project. It remains a mystery to me why using your image will make the mv binary flaky in its behavior. I verified that the mv binary is the same between the two images:


$ podman run --rm --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 --runtime-flag=ignore-cgroups localhost/dev-environment:latest mv --version
mv (GNU coreutils) 9.7
Packaged by Debian (9.7-2)
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Parker, David MacKenzie, and Jim Meyering.

$ podman run --rm --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 --runtime-flag=ignore-cgroups debian:testing mv --version
mv (GNU coreutils) 9.7
Packaged by Debian (9.7-2)
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Parker, David MacKenzie, and Jim Meyering.


    
      
      
    
  


Anyways, I will write a fix for this. I think overlayfs should not pass the RENAME_NOREPLACE to filesystem implementations. This is consistent with Linux; the kernel clears the RENAME_NOREPLACE flag in fs/overlayfs/dir.c:ovl_rename().
ayushr2
ayushr2 commented on Jun 6, 2025 
@ayushr2
ayushr2
on Jun 6, 2025
Collaborator


It remains a mystery to me why using your image will make the mv binary flaky in its behavior. I verified that the mv binary is the same between the two images




I realized that mv is a dynamically linked binary. And confirmed with ldd that their dependency .so files are changing between the two images:


$ podman run --rm --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 --runtime-flag=ignore-cgroups localhost/dev-environment:latest ldd /usr/bin/mv
	linux-vdso.so.1 (0x00007ed280e7b000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007ed280e19000)
	libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007ed280e0e000)
	libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1 (0x00007ed280e06000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ed280c10000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007ed280b61000)
	/lib64/ld-linux-x86-64.so.2 (0x000055ed34c7c000)

$ podman run --rm --name gvisor-bug --user nobody --workdir / --env=LANG=C.UTF-8 --runtime-flag=ignore-cgroups debian:testing ldd /usr/bin/mv
	linux-vdso.so.1 (0x00007edb4c83e000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007edb4c7e2000)
	libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007edb4c7d7000)
	libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1 (0x00007edb4c7cf000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007edb4c5d9000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007edb4c52a000)
	/lib64/ld-linux-x86-64.so.2 (0x0000563baa16a000)


    
      
      
    
  


Notice that the offsets in these files changed. So some dependencies that you are installing in your image is causing the behavior of the mv binary to change.
copybara-service
added a commit that references this issue on Jun 6, 2025
overlayfs: Clear RENAME_NOREPLACE before calling into layers.
9fcf7a6
copybara-service
mentioned this on Jun 6, 2025
dseomn
dseomn commented on Jun 6, 2025 
@dseomn
dseomn
on Jun 6, 2025
Author
I don't think I'm doing anything unusual with mv or any of its dependencies. I tried running debian:testing with an extra apt-get update and apt-get -y dist-upgrade, but that still didn't show the bug. Not sure what else could affect mv.
ayushr2
ayushr2 commented on Jun 7, 2025 
@ayushr2
ayushr2
on Jun 7, 2025
Collaborator
Actually I get this bug on debian:testing too: docker run --runtime=runsc --rm -it debian:testing bash -c 'mkdir /bug-test && touch /bug-test/foo && for i in {1..50}; do echo $i; mv /bug-test/{foo,bar}; mv /bug-test/{bar,foo}; done'


Not sure why with Podman I was not seeing this bug with debian:testing. Anyways, I confirmed that #11798 fixes this issue.
copybara-service
added 4 commits that reference this issue on Jun 9, 2025
overlayfs: Clear RENAME_NOREPLACE before calling into layers.
8f9fe93
overlayfs: Clear RENAME_NOREPLACE before calling into layers.
2eb4e64
overlayfs: Clear RENAME_NOREPLACE before calling into layers.
4e0e7fb
overlayfs: Clear RENAME_NOREPLACE before calling into layers.
cab42c6
copybara-service
closed this as completedin cab42c6on Jun 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Metadata
Assignees
No one assigned
    Labels
    type: bugSomething isn't working
    Type
    No type
    Projects
    No projects
    Milestone
    No milestone
    Relationships
    None yet
      Development
      Participants
      @dseomn@ayushr2
      Issue actions
        
        





  
        

        

    
        
  
        

  
        

          



    



  

    

    

    





    
        
    
        mv prompts about overwriting non-existent files · Issue #11794 · google/gvisor