Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Adding fifo support to fuse4x #25

Open
anatol opened this Issue · 4 comments

2 participants

@anatol
Owner

Hi, Mike.

I am looking into your fifo issue. I can repro the deadlock you described with tup HEAD and fuse4x 0.9.1. Here is the stacktrace:

PID: 7794
    Process: tup
    Thread ID: 0x685c
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0x85c0cb0
    Continuation: psynch_cvcontinue (in mach_kernel) + 0 (0x5c0dd0)
    User stack: 
    __psynch_cvwait (in libsystem_kernel.dylib) + 10 (0x7fff8afa0bca)
      execute_graph (in tup) + 1195 (0x10fe418ff)
        updater (in tup) + 4424 (0x10fe40bbd)
          main (in tup) + 3681 (0x10fe24c29)
            start (in tup) + 52 (0x10fe23dc0)
              0x00000002 (in tup) (0x2)
    Thread ID: 0x685e
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0x10d9f270
    Kernel stack: 
    machine_switch_context (in mach_kernel) + 582 (0x2cca26)
      thread_continue (in mach_kernel) + 1729 (0x22f451)
        thread_block_reason (in mach_kernel) + 331 (0x22f6db)
          thread_block (in mach_kernel) + 33 (0x22f821)
            lck_mtx_sleep (in mach_kernel) + 79 (0x227c7f)
              wakeup (in mach_kernel) + 287 (0x576e9f)
                msleep (in mach_kernel) + 165 (0x5775a5)
                  sbwait (in mach_kernel) + 171 (0x5a89ab)
                    soreceive (in mach_kernel) + 5379 (0x5a4133)
                      pipeinit (in mach_kernel) + 230 (0x588b16)
                        write (in mach_kernel) + 256 (0x586810)
                          read_nocancel (in mach_kernel) + 163 (0x586fa3)
                            unix_syscall64 (in mach_kernel) + 554 (0x5f2fca)
                              lo64_unix_scall (in mach_kernel) + 23 (0x2e5d57)
    User stack: 
    read (in libsystem_kernel.dylib) + 10 (0x7fff8afa1af2)
      child_wait_notifier (in tup) + 125 (0x10fe476b5)
        _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
          thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
    Thread ID: 0x6868
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0xd4602f4
    Kernel stack: 
    machine_switch_context (in mach_kernel) + 582 (0x2cca26)
      thread_continue (in mach_kernel) + 1729 (0x22f451)
        thread_block_reason (in mach_kernel) + 331 (0x22f6db)
          thread_block (in mach_kernel) + 33 (0x22f821)
            lck_mtx_sleep (in mach_kernel) + 79 (0x227c7f)
              wakeup (in mach_kernel) + 287 (0x576e9f)
                msleep (in mach_kernel) + 165 (0x5775a5)
                  fifo_open (in mach_kernel) + 908 (0x33c89c)
                    VNOP_OPEN (in mach_kernel) + 76 (0x32cc3c)
                      vn_open_auth (in mach_kernel) + 1897 (0x322779)
                        mknod (in mach_kernel) + 1344 (0x31ab50)
                          open (in mach_kernel) + 285 (0x31b15d)
                            unix_syscall64 (in mach_kernel) + 554 (0x5f2fca)
                              lo64_unix_scall (in mach_kernel) + 23 (0x2e5d57)
    User stack: 
    __open (in libsystem_kernel.dylib) + 10 (0x7fff8afa0ad2)
      tup_fs_open (in tup) + 157 (0x10fe44db8)
        fuse_compat_open (in libfuse4x.2.dylib) + 44 (0x10fee869c)
          fuse_fs_open (in libfuse4x.2.dylib) + 162 (0x10fee85e2)
            fuse_lib_open (in libfuse4x.2.dylib) + 132 (0x10feee404)
              do_open (in libfuse4x.2.dylib) + 146 (0x10fef65f2)
                fuse_ll_process (in libfuse4x.2.dylib) + 1114 (0x10fef8d1a)
                  fuse_session_process (in libfuse4x.2.dylib) + 75 (0x10fefb3cb)
                    fuse_session_loop (in libfuse4x.2.dylib) + 250 (0x10fef2f0a)
                      fuse_loop (in libfuse4x.2.dylib) + 37 (0x10fef16a5)
                        fuse_thread (in tup) + 16 (0x10fe461aa)
                          _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
                            thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
    Thread ID: 0x6871
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0x9ba0e54
    Continuation: psynch_cvcontinue (in mach_kernel) + 0 (0x5c0dd0)
    User stack: 
    __psynch_cvwait (in libsystem_kernel.dylib) + 10 (0x7fff8afa0bca)
      worker_wait (in tup) + 42 (0x10fe41c91)
        update_work (in tup) + 32 (0x10fe41e2f)
          _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
            thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
    Thread ID: 0x6872
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0xac5c038
    Continuation: psynch_cvcontinue (in mach_kernel) + 0 (0x5c0dd0)
    User stack: 
    __psynch_cvwait (in libsystem_kernel.dylib) + 10 (0x7fff8afa0bca)
      worker_wait (in tup) + 42 (0x10fe41c91)
        update_work (in tup) + 32 (0x10fe41e2f)
          _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
            thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
    Thread ID: 0x6873
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0x6bdc604
    Continuation: psynch_cvcontinue (in mach_kernel) + 0 (0x5c0dd0)
    User stack: 
    __psynch_cvwait (in libsystem_kernel.dylib) + 10 (0x7fff8afa0bca)
      worker_wait (in tup) + 42 (0x10fe41c91)
        update_work (in tup) + 32 (0x10fe41e2f)
          _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
            thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
    Thread ID: 0x6874
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0xb27ed10
    Continuation: psynch_cvcontinue (in mach_kernel) + 0 (0x5c0dd0)
    User stack: 
    __psynch_cvwait (in libsystem_kernel.dylib) + 10 (0x7fff8afa0bca)
      master_fork_exec (in tup) + 393 (0x10fe479a1)
        exec_internal (in tup) + 417 (0x10fe4649e)
          server_exec (in tup) + 80 (0x10fe462d8)
            update_work (in tup) + 1045 (0x10fe42224)
              _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
                thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
PID: 7795
    Process: tup
    Thread ID: 0x685d
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0xab141a4
    Kernel stack: 
    machine_switch_context (in mach_kernel) + 582 (0x2cca26)
      thread_continue (in mach_kernel) + 1729 (0x22f451)
        thread_block_reason (in mach_kernel) + 331 (0x22f6db)
          thread_block (in mach_kernel) + 33 (0x22f821)
            lck_mtx_sleep (in mach_kernel) + 79 (0x227c7f)
              wakeup (in mach_kernel) + 287 (0x576e9f)
                msleep (in mach_kernel) + 165 (0x5775a5)
                  sbwait (in mach_kernel) + 171 (0x5a89ab)
                    soreceive (in mach_kernel) + 5379 (0x5a4133)
                      pipeinit (in mach_kernel) + 230 (0x588b16)
                        write (in mach_kernel) + 256 (0x586810)
                          read_nocancel (in mach_kernel) + 163 (0x586fa3)
                            unix_syscall64 (in mach_kernel) + 554 (0x5f2fca)
                              lo64_unix_scall (in mach_kernel) + 23 (0x2e5d57)
    User stack: 
    read (in libsystem_kernel.dylib) + 10 (0x7fff8afa1af2)
      server_pre_init (in tup) + 767 (0x10fe46f13)
        tup_init (in tup) + 52 (0x10fe36270)
          main (in tup) + 1531 (0x10fe243c3)
            start (in tup) + 52 (0x10fe23dc0)
              0x00000002 (in tup) (0x2)
    Thread ID: 0x6876
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0x87720a0
    Continuation: wakeup (in mach_kernel) + 1024 (0x577180)
    User stack: 
    __wait4 (in libsystem_kernel.dylib) + 10 (0x7fff8afa1156)
      _pthread_start (in libsystem_c.dylib) + 335 (0x7fff946a08bf)
        thread_start (in libsystem_c.dylib) + 13 (0x7fff946a3b75)
PID: 7797
    Process: sh
    Thread ID: 0x6875
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0xdcf3b00
    Kernel stack: 
    machine_switch_context (in mach_kernel) + 582 (0x2cca26)
      thread_continue (in mach_kernel) + 1729 (0x22f451)
        thread_block_reason (in mach_kernel) + 331 (0x22f6db)
          thread_block (in mach_kernel) + 33 (0x22f821)
            lck_mtx_sleep_deadline (in mach_kernel) + 93 (0x227bad)
              wakeup (in mach_kernel) + 259 (0x576e83)
                msleep (in mach_kernel) + 165 (0x5775a5)
fuse_ticket_wait_answer (in org.fuse4x.kext.fuse4x.sym) (fuse_ipc.c:262)
(0xf9734d)
fuse_dispatcher_wait_answer (in org.fuse4x.kext.fuse4x.sym) (fuse_ipc.c:976)
(0xf98be0)
fuse_filehandle_get (in org.fuse4x.kext.fuse4x.sym) (fuse_file.c:79)
(0xf932bf)
fuse_vnop_open (in org.fuse4x.kext.fuse4x.sym) (fuse_vnops.c:1769)
(0xfa1770)
                          VNOP_OPEN (in mach_kernel) + 76 (0x32cc3c)
                            vn_open_auth (in mach_kernel) + 1897 (0x322779)
                              mknod (in mach_kernel) + 1344 (0x31ab50)
                                open (in mach_kernel) + 285 (0x31b15d)
                                  unix_syscall64 (in mach_kernel) + 554 (0x5f2fca)
                                    lo64_unix_scall (in mach_kernel) + 23 (0x2e5d57)
    User stack: 
    __open (in libsystem_kernel.dylib) + 10 (0x7fff8afa0ad2)
      0x000000010525550e (in sh) + 98 (0x10525550e)
        0x000000010523c684 (in sh) + 5594 (0x10523c684)
          0x000000010523b04f (in sh) + 95 (0x10523b04f)
            0x0000000105239e5e (in sh) + 474 (0x105239e5e)
              0x0000000105220307 (in sh) + 6286 (0x105220307)
                0x000000010521e3e8 (in sh) + 52 (0x10521e3e8)
                  0x00000002 (in sh) (0x2)
PID: 7799
    Process: sh
    Thread ID: 0x687d
    Thread state: 0x1 == TH_WAIT 
    Thread wait_event: 0xdb18200
    Kernel stack: 
    machine_switch_context (in mach_kernel) + 582 (0x2cca26)
      thread_continue (in mach_kernel) + 1729 (0x22f451)
        thread_block_reason (in mach_kernel) + 331 (0x22f6db)
          thread_block (in mach_kernel) + 33 (0x22f821)
            lck_mtx_sleep_deadline (in mach_kernel) + 93 (0x227bad)
              wakeup (in mach_kernel) + 259 (0x576e83)
                msleep (in mach_kernel) + 165 (0x5775a5)
fuse_ticket_wait_answer (in org.fuse4x.kext.fuse4x.sym) (fuse_ipc.c:262)
(0xf9734d)
fuse_dispatcher_wait_answer (in org.fuse4x.kext.fuse4x.sym) (fuse_ipc.c:976)
(0xf98be0)
fuse_internal_access (in org.fuse4x.kext.fuse4x.sym) (fuse_internal.c:129)
(0xf938df)
fuse_vnop_access (in org.fuse4x.kext.fuse4x.sym) (fuse_vnops.c:103)
(0xf9d7e6)
                          VNOP_ACCESS (in mach_kernel) + 96 (0x32ca90)
                            vn_attribute_prepare (in mach_kernel) + 992 (0x3086f0)
                              kauth_authorize_action (in mach_kernel) + 85 (0x54c5e5)
                                vnode_authorize (in mach_kernel) + 94 (0x3079be)
                                  vn_stat (in mach_kernel) + 66 (0x321c62)
                                    pathconf (in mach_kernel) + 370 (0x3164b2)
                                      stat64 (in mach_kernel) + 200 (0x317128)
                                        unix_syscall64 (in mach_kernel) + 554 (0x5f2fca)
                                          lo64_unix_scall (in mach_kernel) + 23 (0x2e5d57)
    User stack: 
    stat$INODE64 (in libsystem_kernel.dylib) + 10 (0x7fff8afa1e9e)
      0x0000000105240fe5 (in sh) + 251 (0x105240fe5)
        0x000000010523ca3b (in sh) + 6545 (0x10523ca3b)
          0x000000010523df30 (in sh) + 11910 (0x10523df30)
            0x000000010523b04f (in sh) + 95 (0x10523b04f)
              0x0000000105239e5e (in sh) + 474 (0x105239e5e)
                0x0000000105220307 (in sh) + 6286 (0x105220307)
                  0x000000010521e3e8 (in sh) + 52 (0x10521e3e8)
                    0x00000002 (in sh) (0x2)

I see 2 shell processes that wait response from userspace daemon. I also see that the userspace is blocked in open(), this is most likely it expects when the other side of fifo will be opened. This makes me believe that tup fuse fs runs in a single-thread mode.

Does it ring the bell?

@anatol
Owner

@gittup

Actually I use slightly modified version of fuse4x kext. Let me know if you want to repro this deadlock on your computer - I'll provide you instructions how to build the kext.

@gittup

Yes, tup is currently running fuse in a single thread. This can be made multi-threaded by calling fuse_loop_mt() instead of fuse_loop() in src/tup/server/fuse_server.c if necessary. Was the issue in the example file-system already addressed, and now it just doesn't work in tup? http://fuse4x.1054701.n5.nabble.com/fifos-td5639828.html

@anatol
Owner

That test in your message still does not work in HEAD but I work on it.

I just was surprised how tup works in Linux in single-thread mode. Fifo needs 2 simultaneous fuse requests to userspace and while one of them is blocked, another opens the same file from other thread. It implied that tup should use multithread mode. Did I miss something?

@gittup

Hmm, that is quite strange. Maybe in the tup test when the grep process tries to read from the fifo, it is blocking in the kernel before it gets to the fuse fs? Otherwise I am not sure why it would work. I don't think there is any reason tup can't run a multi-threaded fuse - it is just single threaded to avoid innocuous reports from helgrind. If the fifo test (outside of tup) can run on fuse4x, we can see if tup will have an issue with it in single-threaded mode or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.