Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nbdkit server hangs during negotiation when running in background, works in foreground #16

Closed
nathanshearer opened this issue Oct 24, 2021 · 1 comment
Closed

nbdkit server hangs during negotiation when running in background, works in foreground #16

nathanshearer opened this issue Oct 24, 2021 · 1 comment

Comments

@nathanshearer
Copy link

With my current setup, the issue is reproducible. If I start the nbdkit server by placing a startup script in /etc/local.d to start on boot, it will start, but the nbd-client will always hang during negotiation. Killing and restarting the server and/or the client does not resolve the deadlock.

However, if I run the exact same script manually, by logging into the server, then nbd-client is able to successfully connect to the nbdkit server and everything works as expected.

During my tests, I found that when I killed nbdkit, it would see errors about their being insufficient "magic" or an invalid password. I am not using any passwords, and both the nbdkit server and nbd-client are on the same host, connecting through 127.0.0.1
@rwmjones
Copy link
Member

This site is a mirror of the https://gitlab.com/nbdkit/nbdkit

I followed up with some questions on the mailing list: https://listman.redhat.com/archives/libguestfs/2021-October/msg00120.html

libguestfs pushed a commit that referenced this issue Oct 25, 2021
@rwmjones
tests: vddk: Disable GLIBC_TUNABLES for VDDK tests
3c44670 
VDDK 7.0.0 and 7.0.3 crash with GLIBC_TUNABLES (malloc debugging) and
latest glibc.  This likely indicates a bug in VDDK, but as we cannot
fix it we need to disable the extra hygiene checks instead.  For
information the stack trace is below.

  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  39	  magic = (((uintptr_t) p >> 3) ^ ((uintptr_t) p >> 11)) & 0xFF;
  [Current thread is 1 (Thread 0x7fea8056b640 (LWP 3905172))]
  (gdb) bt
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  #1  malloc_usable_size (mem=0x0) at malloc-debug.c:405
  #2  0x00007fea7fd5199b in greedy_realloc0 (p=p@entry=0x7fea805690c0,
      need=need@entry=2, size=size@entry=16) at ../src/basic/alloc-util.c:87
  #3  0x00007fea7fd51b13 in netlink_container_parse (m=m@entry=0x555d34e77f80,
      container=0x555d34e77f90, rta=0x555d34e821f8, rt_len=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-message.c:1078
  #4  0x00007fea7fd557ec in sd_netlink_message_rewind (genl=<optimized out>,
      m=<optimized out>) at ../src/libsystemd/sd-netlink/netlink-message.c:1324
  #5  socket_read_message (rtnl=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-socket.c:375
  #6  sd_netlink_read (usec=0, ret=0x7fea805692a0, serial=<optimized out>,
      rtnl=<optimized out>) at ../src/libsystemd/sd-netlink/sd-netlink.c:700
  #7  sd_netlink_call (rtnl=<optimized out>, message=<optimized out>,
      ret=0x7fea805692a0, usec=0)
      at ../src/libsystemd/sd-netlink/sd-netlink.c:743
  #8  0x00007fea7fd5818e in local_addresses (af=2, ret=0x7fea80569378,
      ifindex=0, context=0x0) at ../src/shared/local-addresses.c:90
  #9  0x00007fea7fd590b9 in _nss_myhostname_gethostbyname3_r (
      name=<optimized out>, af=<optimized out>, host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594, ttlp=0x0, canonp=0x0)
      at ../src/nss-myhostname/nss-myhostname.c:373
  #10 0x00007fea7fd593cb in _nss_myhostname_gethostbyname_r (
      name=0x7fea80569601 "pick", host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594) at ../src/nss-myhostname/nss-myhostname.c:514
  #11 0x00007fea864491cb in __gethostbyname_r (
      name=0x7fea80569601 "pick", resbuf=0x7fea805695a0,
      buffer=<optimized out>, buflen=1024, result=<optimized out>,
      h_errnop=0x7fea80569594) at ../nss/getXXbyYY_r.c:273
  #12 0x00007fea85bdcca4 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #13 0x00007fea85bd2bb2 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #14 0x00007fea85bba567 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #15 0x00007fea85bbab75 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #16 0x00007fea85bc63bd in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #17 0x00007fea85bc42ec in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #18 0x00007fea85bbf026 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #19 0x00007fea85bc0038 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #20 0x00007fea85c2fb23 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #21 0x00007fea85c2f160 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #22 0x00007fea85c2f1ba in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #23 0x00007fea85d0c0b9 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #24 0x00007fea85d079cb in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #25 0x00007fea85cb2831 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #26 0x00007fea85ca4702 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #27 0x00007fea85c9f76f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #28 0x00007fea85c7b59f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #29 0x00007fea85c7c11d in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #30 0x00007fea85bb4475 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #31 0x00007fea85bb4717 in VixDiskLib_Open ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #32 0x00007fea867b0dbb in vddk_open (readonly=<optimized out>)
      at /home/rjones/d/nbdkit/plugins/vddk/vddk.c:717
  #33 0x0000555d2a83d922 in plugin_open (c=0x555d34e81d10, readonly=0,
      exportname=<optimized out>, is_tls=<optimized out>)
      at /home/rjones/d/nbdkit/server/plugins.c:353
  #34 0x0000555d2a83572d in backend_open (b=0x555d34668db0, readonly=0,
      exportname=0x555d34dae020 "", shared=0)
      at /home/rjones/d/nbdkit/server/backend.c:292
  #35 0x0000555d2a83b937 in protocol_common_open (
      exportsize=exportsize@entry=0x7fea8056a9e0,
      flags=flags@entry=0x555d34e7f604,
      exportname=exportname@entry=0x555d34e7eeb0 "")
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:83
  #36 0x0000555d2a83e140 in finish_newstyle_options (exportsize=0x7fea8056a9e0,
      exportname_in=<optimized out>, exportnamelen=<optimized out>)
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:284
  #37 0x0000555d2a837ee0 in negotiate_handshake_newstyle_options ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:572
  #38 protocol_handshake_newstyle ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:904
  #39 protocol_handshake ()
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:55
  #40 handle_single_connection (sockin=<optimized out>, sockout=<optimized out>)
      at /home/rjones/d/nbdkit/server/connections.c:171
  #41 0x0000555d2a83e4e0 in start_thread (datav=0x555d34d987d0)
      at /home/rjones/d/nbdkit/server/sockets.c:354
  #42 0x00007fea863b4677 in start_thread (arg=<optimized out>)
      at pthread_create.c:434
  #43 0x00007fea86439320 in clone3 ()
      at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

(cherry picked from commit 03406a7)
rwmjones added a commit to rwmjones/nbdkit that referenced this issue Oct 28, 2021
@rwmjones
tests: vddk: Disable GLIBC_TUNABLES for VDDK tests
03406a7 
VDDK 7.0.0 and 7.0.3 crash with GLIBC_TUNABLES (malloc debugging) and
latest glibc.  This likely indicates a bug in VDDK, but as we cannot
fix it we need to disable the extra hygiene checks instead.  For
information the stack trace is below.

  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  39	  magic = (((uintptr_t) p >> 3) ^ ((uintptr_t) p >> 11)) & 0xFF;
  [Current thread is 1 (Thread 0x7fea8056b640 (LWP 3905172))]
  (gdb) bt
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  libguestfs#1  malloc_usable_size (mem=0x0) at malloc-debug.c:405
  libguestfs#2  0x00007fea7fd5199b in greedy_realloc0 (p=p@entry=0x7fea805690c0,
      need=need@entry=2, size=size@entry=16) at ../src/basic/alloc-util.c:87
  libguestfs#3  0x00007fea7fd51b13 in netlink_container_parse (m=m@entry=0x555d34e77f80,
      container=0x555d34e77f90, rta=0x555d34e821f8, rt_len=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-message.c:1078
  libguestfs#4  0x00007fea7fd557ec in sd_netlink_message_rewind (genl=<optimized out>,
      m=<optimized out>) at ../src/libsystemd/sd-netlink/netlink-message.c:1324
  libguestfs#5  socket_read_message (rtnl=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-socket.c:375
  libguestfs#6  sd_netlink_read (usec=0, ret=0x7fea805692a0, serial=<optimized out>,
      rtnl=<optimized out>) at ../src/libsystemd/sd-netlink/sd-netlink.c:700
  libguestfs#7  sd_netlink_call (rtnl=<optimized out>, message=<optimized out>,
      ret=0x7fea805692a0, usec=0)
      at ../src/libsystemd/sd-netlink/sd-netlink.c:743
  libguestfs#8  0x00007fea7fd5818e in local_addresses (af=2, ret=0x7fea80569378,
      ifindex=0, context=0x0) at ../src/shared/local-addresses.c:90
  libguestfs#9  0x00007fea7fd590b9 in _nss_myhostname_gethostbyname3_r (
      name=<optimized out>, af=<optimized out>, host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594, ttlp=0x0, canonp=0x0)
      at ../src/nss-myhostname/nss-myhostname.c:373
  libguestfs#10 0x00007fea7fd593cb in _nss_myhostname_gethostbyname_r (
      name=0x7fea80569601 "pick", host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594) at ../src/nss-myhostname/nss-myhostname.c:514
  libguestfs#11 0x00007fea864491cb in __gethostbyname_r (
      name=0x7fea80569601 "pick", resbuf=0x7fea805695a0,
      buffer=<optimized out>, buflen=1024, result=<optimized out>,
      h_errnop=0x7fea80569594) at ../nss/getXXbyYY_r.c:273
  libguestfs#12 0x00007fea85bdcca4 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#13 0x00007fea85bd2bb2 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#14 0x00007fea85bba567 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#15 0x00007fea85bbab75 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#16 0x00007fea85bc63bd in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#17 0x00007fea85bc42ec in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#18 0x00007fea85bbf026 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#19 0x00007fea85bc0038 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#20 0x00007fea85c2fb23 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#21 0x00007fea85c2f160 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #22 0x00007fea85c2f1ba in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #23 0x00007fea85d0c0b9 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #24 0x00007fea85d079cb in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #25 0x00007fea85cb2831 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #26 0x00007fea85ca4702 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #27 0x00007fea85c9f76f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #28 0x00007fea85c7b59f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #29 0x00007fea85c7c11d in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #30 0x00007fea85bb4475 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #31 0x00007fea85bb4717 in VixDiskLib_Open ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #32 0x00007fea867b0dbb in vddk_open (readonly=<optimized out>)
      at /home/rjones/d/nbdkit/plugins/vddk/vddk.c:717
  #33 0x0000555d2a83d922 in plugin_open (c=0x555d34e81d10, readonly=0,
      exportname=<optimized out>, is_tls=<optimized out>)
      at /home/rjones/d/nbdkit/server/plugins.c:353
  #34 0x0000555d2a83572d in backend_open (b=0x555d34668db0, readonly=0,
      exportname=0x555d34dae020 "", shared=0)
      at /home/rjones/d/nbdkit/server/backend.c:292
  #35 0x0000555d2a83b937 in protocol_common_open (
      exportsize=exportsize@entry=0x7fea8056a9e0,
      flags=flags@entry=0x555d34e7f604,
      exportname=exportname@entry=0x555d34e7eeb0 "")
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:83
  #36 0x0000555d2a83e140 in finish_newstyle_options (exportsize=0x7fea8056a9e0,
      exportname_in=<optimized out>, exportnamelen=<optimized out>)
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:284
  #37 0x0000555d2a837ee0 in negotiate_handshake_newstyle_options ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:572
  #38 protocol_handshake_newstyle ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:904
  #39 protocol_handshake ()
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:55
  #40 handle_single_connection (sockin=<optimized out>, sockout=<optimized out>)
      at /home/rjones/d/nbdkit/server/connections.c:171
  #41 0x0000555d2a83e4e0 in start_thread (datav=0x555d34d987d0)
      at /home/rjones/d/nbdkit/server/sockets.c:354
  #42 0x00007fea863b4677 in start_thread (arg=<optimized out>)
      at pthread_create.c:434
  #43 0x00007fea86439320 in clone3 ()
      at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
rwmjones added a commit to rwmjones/nbdkit that referenced this issue Aug 16, 2023
@rwmjones
Merge branch '2023-rust-block-size' into 'master'
4f08068 
rust: Add binding for .block_size

Closes libguestfs#16

See merge request nbdkit/nbdkit!34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rwmjones @nathanshearer