Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestFUSEOverlayFS/LayerFileupdate fails depending on the number of CPUs #2

Closed
AkihiroSuda opened this issue Oct 20, 2019 · 7 comments
Closed

Comments

@AkihiroSuda
Copy link
Member

$ make test
...
    --- PASS: TestFUSEOverlayFS/StatActive (0.00s)                                                                                                                                                                                                                                                                                            
        helpers_unix.go:32: unmount /tmp/snapshot-suite-fuse-overlayfs-355131582/work/preparing                                                                                                                                                                                                                                               
    --- FAIL: TestFUSEOverlayFS/LayerFileupdate (0.57s)                                                                                                                                                                                                                                                                                       
        issues.go:62: Check snapshots failed: directory diff between /tmp/snapshot-suite-fuse-overlayfs-538511347/work/flat975402205 and /tmp/snapshot-suite-fuse-overlayfs-538511347/work/check160206373                                                                                                                                     
            ~ /root/.bashrc(mode: 644, uid: 0, gid: 0) -> /root/.bashrc(mode: 644, uid: 0, gid: 0)                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                              
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/continuity/fs/fstest.CheckDirectoryEqual                                                                                                                                                                                                            
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/continuity/fs/fstest/compare.go:52                                                                                                                                                                                                      
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkSnapshot                                                                                                                                                                                                        
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/helpers.go:104                                                                                                                                                                                           
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkSnapshots                                                                                                                                                                                                       
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/helpers.go:132                                                                                                                                                                                           
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkLayerFileUpdate                                                                                                                                                                                                 
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/issues.go:61                                                                                                                                                                                             
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.makeTest.func1                                                                                                                                                                                                       
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/testsuite.go:112                                                                                                                                                                                         
            testing.tRunner                                                                                                                                                                                                                                                                                                                   
                /usr/local/go/src/testing/testing.go:909                                                                                                                                                                                                                                                                                      
            runtime.goexit                                                                                                                                                                                                                                                                                                                    
                /usr/local/go/src/runtime/asm_amd64.s:1357                                                                                                                                                                                                                                                                                    
            check directory failed                                                                                                                                                                                                                                                                                                            
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkSnapshot                                                                                                                                                                                                        
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/helpers.go:105                                                                                                                                                                                           
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkSnapshots                                                                                                                                                                                                       
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/helpers.go:132                                                                                                                                                                                           
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkLayerFileUpdate                                                                                                                                                                                                 
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/issues.go:61                                                                                                                                                                                             
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.makeTest.func1                                                                                                                                                                                                       
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/testsuite.go:112                                                                                                                                                                                         
            testing.tRunner                                                                                                                                                                                                                                                                                                                   
                /usr/local/go/src/testing/testing.go:909                                                                                                                                                                                                                                                                                      
            runtime.goexit                                                                                                                                                                                                                                                                                                                    
                /usr/local/go/src/runtime/asm_amd64.s:1357                                                                                                                                                                                                                                                                                    
            snapshot check failed on snapshot 2                                                                                                                                                                                                                                                                                               
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkSnapshots                                                                                                                                                                                                       
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/helpers.go:133                                                                                                                                                                                           
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.checkLayerFileUpdate                                                                                                                                                                                                 
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/issues.go:61                                                                                                                                                                                             
            github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite.makeTest.func1                                                                                                                                                                                                       
                /go/src/github.com/AkihiroSuda/containerd-fuse-overlayfs/vendor/github.com/containerd/containerd/snapshots/testsuite/testsuite.go:112                                                                                                                                                                                         
            testing.tRunner                                                                                                                                                                                                                                                                                                                   
                /usr/local/go/src/testing/testing.go:909                                                                                                                                                                                                                                                                                      
            runtime.goexit                                                                                                                                                                                                                                                                                                                    
                /usr/local/go/src/runtime/asm_amd64.s:1357                                                                                                                                                                                                                                                                                    
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347                                                                                                                                                                                                                                                     
        helpers.go:67: drwxr-xr-x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root                                                                                                                                                                                                                                                
        helpers.go:65: -rw-------      65536 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/metadata.db [ "\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\xed\xda\f\xed\x02\x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\b\x00\x00\x00\x00\x00\x00\x00\x0f\x
00\x00\x00\x00\x00\x00\x00" ...]                                                                                                                                                                                                                                                                                                              
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots                                                                                                                                                                                                                                      
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/1                                                                                                                                                                                                                                    
        helpers.go:67: drwxr-xr-x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/1/fs                                                                                                                                                                                                                                 
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/1/fs/etc                                                                                                                                                                                                                             
        helpers.go:65: -rw-r--r--         17 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/1/fs/etc/hosts [ "mydomain 10.0.0.1" ...]                                                                                                                                                                                            
        helpers.go:65: -rw-r--r--         13 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/1/fs/etc/profile [ "PATH=/usr/bin" ...]                                                                                                                                                                                              
        helpers.go:67: drwx--x--x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/1/work                                                                                                                                                                                                                               
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3                                                                                                                                                                                                                                    
        helpers.go:67: drwxr-xr-x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/fs                                                                                                                                                                                                                                 
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/fs/etc                                                                                                                                                                                                                             
        helpers.go:65: -rw-r--r--         17 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/fs/etc/hosts [ "mydomain 10.0.0.2" ...]                                                                                                                                                                                            
        helpers.go:65: -rw-rw-rw-         13 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/fs/etc/profile [ "PATH=/usr/bin" ...]                                                                                                                                                                                              
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/fs/root                                                                                                                                                                                                                            
        helpers.go:65: -rw-r--r--         23 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/fs/root/.bashrc [ "PATH=/usr/sbin:/usr/bin" ...]                                                                                                                                                                                   
        helpers.go:67: drwx--x--x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/work                                                                                                                                                                                                                               
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/3/work/work                                                                                                                                                                                                                          
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/5                                                                                                                                                                                                                                    
        helpers.go:67: drwxr-xr-x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/5/fs                                                                                                                                                                                                                                 
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/5/fs/etc                                                                                                                                                                                                                             
        helpers.go:65: -rw-r--r--         17 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/5/fs/etc/hosts [ "mydomain 10.0.0.1" ...]                                                                                                                                                                                            
        helpers.go:65: -rw-r--r--         13 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/5/fs/etc/profile [ "PATH=/usr/bin" ...]                                                                                                                                                                                              
        helpers.go:67: drwx--x--x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/5/work                                                                                                                                                                                                                               
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7                                                                                                                                                                                                                                    
        helpers.go:67: drwxr-xr-x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/fs                                                                                                                                                                                                                                 
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/fs/etc                                                                                                                                                                                                                             
        helpers.go:65: -rw-r--r--         17 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/fs/etc/hosts [ "mydomain 10.0.0.2" ...]                                                                                                                                                                                            
        helpers.go:65: -rw-rw-rw-         13 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/fs/etc/profile [ "PATH=/usr/bin" ...]                                                                                                                                                                                              
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/fs/root                                                                                                                                                                                                                            
        helpers.go:65: -rw-r--r--          0 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/fs/root/.bashrc [ "" ...]                                                                                                                                                                                                          
        helpers.go:67: drwx--x--x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/work                                                                                                                                                                                                                               
        helpers.go:67: drwx------       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/root/snapshots/7/work/work                                                                                                                                                                                                                          
        helpers.go:67: drwxr-xr-x       4096 /tmp/snapshot-suite-fuse-overlayfs-538511347/work                                                                                                                                                                                                                                                
    --- PASS: TestFUSEOverlayFS/RootPermission (0.00s)                                                                                                                 
        helpers_unix.go:32: unmount /tmp/snapshot-suite-fuse-overlayfs-981504114/work/preparing    

tree: 5e1e2cf
kernel: 5.0.0-31-generic #33~18.04.1-Ubuntu


make test succeeds on Ubuntu 19.10 kernel 5.3.0-18-generic #19-Ubuntu

@AkihiroSuda
Copy link
Member Author

fails on Ubuntu 19.04 kernel 5.0.0-31-generic #33-Ubuntu as well

@AkihiroSuda
Copy link
Member Author

It fails on my another Ubuntu 19.10 machine. Maybe unrelated to kernel version.

@AkihiroSuda
Copy link
Member Author

containerd-fuse-overlayfs.test -test.root -test.v -test.run TestFUSEOverlayFS/LayerFileupdate fails even when running as the root on some hosts

However, it succeeds when running with strace -ff ... 🤔

Maybe timestamp issue

@AkihiroSuda
Copy link
Member Author

@giuseppe

This seems concurrrency issue.

This issue doesn't happen on GCE n1-standard-4, but when I add taskset 0x1, the issue happens.

@AkihiroSuda
Copy link
Member Author

fuse-overlayfs -s doesn't solve the issue

@AkihiroSuda AkihiroSuda changed the title make test fails on Ubuntu 18.04 HWE kernel 5.0 TestFUSEOverlayFS/LayerFileupdate fails depending on the number of CPUs Oct 21, 2019
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Oct 21, 2019

"updated" file has size:0

$ git diff
diff --git a/vendor/github.com/containerd/continuity/fs/fstest/continuity_util.go b/vendor/github.com/containerd/continuity/fs/fstest/continuity_util.go
index 4d30dd0..03a8729 100644
--- a/vendor/github.com/containerd/continuity/fs/fstest/continuity_util.go
+++ b/vendor/github.com/containerd/continuity/fs/fstest/continuity_util.go
@@ -29,10 +29,11 @@ type resourceUpdate struct {
 }
 
 func (u resourceUpdate) String() string {
-       return fmt.Sprintf("%s(mode: %o, uid: %d, gid: %d) -> %s(mode: %o, uid: %d, gid: %d)",
-               u.Original.Path(), u.Original.Mode(), u.Original.UID(), u.Original.GID(),
-               u.Updated.Path(), u.Updated.Mode(), u.Updated.UID(), u.Updated.GID(),
-       )
+       return fmt.Sprintf("%s(%+v) -> %s(%+v)", u.Original.Path(), u.Original, u.Updated.Path(), u.Updated)
+//     return fmt.Sprintf("%s(mode: %o, uid: %d, gid: %d) -> %s(mode: %o, uid: %d, gid: %d)",
+//             u.Original.Path(), u.Original.Mode(), u.Original.UID(), u.Original.GID(),
+//             u.Updated.Path(), u.Updated.Mode(), u.Updated.UID(), u.Updated.GID(),
+//     )
 }
 
 type resourceListDifference struct {

$ go test -c . && sudo taskset --cpu-list 1 ./containerd-fuse-overlayfs.t
est -test.root -test.v -test.run TestFUSEOverlayFS/LayerFileupdate                                                                                     
=== RUN   TestFUSEOverlayFS                                                                                                                            
=== RUN   TestFUSEOverlayFS/LayerFileupdate                                                                                                            
=== PAUSE TestFUSEOverlayFS/LayerFileupdate                                                                                                            
=== CONT  TestFUSEOverlayFS/LayerFileupdate                                                                                                            
--- FAIL: TestFUSEOverlayFS (0.01s)                                                                                                                    
    --- FAIL: TestFUSEOverlayFS/LayerFileupdate (0.04s)                                                                                                
        issues.go:62: Check snapshots failed: directory diff between /tmp/snapshot-suite-fuse-overlayfs-823498837/work/flat106545072 and /tmp/snapshot-
suite-fuse-overlayfs-823498837/work/check314936477                                                                                                     
            ~ /root/.bashrc(&{resource:{paths:[/root/.bashrc] mode:420 uid:0 gid:0 xattrs:map[]} size:23 digests:[sha256:a28a732b0fa5e1774f8ff746277d44
e5796d523fe5312789d9c5b87f4824f596]}) -> /root/.bashrc(&{resource:{paths:[/root/.bashrc] mode:420 uid:0 gid:0 xattrs:map[]} size:0 digests:[sha256:e3b0
c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]})
...

@AkihiroSuda
Copy link
Member Author

Mounting fuse-overlayfs with writeback=0 seems to solve the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant