Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nbdkit logs to stderr when "--verbose" flag given even without "--foreground" flag #18

Closed
archiecobbs opened this issue Jul 8, 2022 · 2 comments

Comments

@archiecobbs
Copy link

Versions:

$ rpm -qa|grep nbd|sort
nbd-3.24-150000.3.3.1.x86_64
nbdkit-1.29.4-150300.8.6.1.x86_64
nbdkit-basic-filters-1.29.4-150300.8.6.1.x86_64
nbdkit-basic-plugins-1.29.4-150300.8.6.1.x86_64
nbdkit-debuginfo-1.29.4-150300.8.6.1.x86_64
nbdkit-devel-1.29.4-150300.8.6.1.x86_64
nbdkit-server-1.29.4-150300.8.6.1.x86_64
nbdkit-server-debuginfo-1.29.4-150300.8.6.1.x86_64

Not sure if I'm doing something wrong, but it seems like logging to stderr works too well, and logging to syslog completely does not work.

Problem 1: when the --verbose flag is given to nbdkit(1), it logs to stderr (a) even though the --foreground flag is not given. This happens even if the --log=syslog flag is given.

Here's an example. When my program (s3backer) runs with the --debug flag, it fork()'s off nbdkit(1) as shown in the log below with the --verbose flag. But then nbdkit(1) continues to log stuff to syslog even after it's forked into the background:

$ ./s3backer --nbd --debug --listBlocks --readOnly s3backer-demo /dev/nbd1
2022-07-08 12:42:57 INFO: s3backer: executing /usr/sbin/nbdkit with these parameters:
2022-07-08 12:42:57 INFO: s3backer:   [00] "/usr/sbin/nbdkit"
2022-07-08 12:42:57 INFO: s3backer:   [01] "--verbose"
2022-07-08 12:42:57 INFO: s3backer:   [02] "--read-only"
2022-07-08 12:42:57 INFO: s3backer:   [03] "--log=syslog"
2022-07-08 12:42:57 INFO: s3backer:   [04] "--filter=exitlast"
2022-07-08 12:42:57 INFO: s3backer:   [05] "--unix"
2022-07-08 12:42:57 INFO: s3backer:   [06] "/var/run/s3backer-nbd/0000000000000006_0000000000000538"
2022-07-08 12:42:57 INFO: s3backer:   [07] "s3backer"
2022-07-08 12:42:57 INFO: s3backer:   [08] "s3b_debug=true"
2022-07-08 12:42:57 INFO: s3backer:   [09] "s3b_listBlocks=true"
2022-07-08 12:42:57 INFO: s3backer:   [10] "s3b_readOnly=true"
2022-07-08 12:42:57 INFO: s3backer:   [11] "bucket=s3backer-demo"
2022-07-08 12:42:57 INFO: s3backer: started /usr/sbin/nbdkit as process 29690
nbdkit: debug: nbdkit 1.29.4 (nbdkit-1.29.4)
nbdkit: debug: TLS disabled: could not load TLS certificates
nbdkit: debug: registering plugin /usr/lib64/nbdkit/plugins/nbdkit-s3backer-plugin.so
nbdkit: debug: registered plugin /usr/lib64/nbdkit/plugins/nbdkit-s3backer-plugin.so (name s3backer)
nbdkit: debug: s3backer: load
nbdkit: debug: registering filter /usr/lib64/nbdkit/filters/nbdkit-exitlast-filter.so
nbdkit: debug: registered filter /usr/lib64/nbdkit/filters/nbdkit-exitlast-filter.so (name exitlast)
nbdkit: debug: exitlast: load
nbdkit: debug: exitlast: config key=s3b_debug, value=true
nbdkit: debug: s3backer: config key=s3b_debug, value=true
nbdkit: debug: exitlast: config key=s3b_listBlocks, value=true
nbdkit: debug: s3backer: config key=s3b_listBlocks, value=true
nbdkit: debug: exitlast: config key=s3b_readOnly, value=true
nbdkit: debug: s3backer: config key=s3b_readOnly, value=true
nbdkit: debug: exitlast: config key=bucket, value=s3backer-demo
nbdkit: debug: s3backer: config key=bucket, value=s3backer-demo
nbdkit: debug: exitlast: config_complete
nbdkit: debug: s3backer: config_complete
nbdkit: debug: recording bucket parameter "s3backer-demo"
...
nbdkit: debug: accepted connection
nbdkit: s3backer[1]: debug: exitlast: preconnect
nbdkit: s3backer[1]: debug: s3backer: preconnect
nbdkit: s3backer[1]: debug: newstyle negotiation: flags: global 0x3
nbdkit: s3backer[1]: debug: newstyle negotiation: client flags: 0x3
nbdkit: s3backer[1]: debug: newstyle negotiation: NBD_OPT_GO: client requested export ''
nbdkit: s3backer[1]: debug: exitlast: open readonly=1 exportname="" tls=0
nbdkit: s3backer[1]: debug: exitlast: default_export readonly=1 tls=0
nbdkit: s3backer[1]: debug: s3backer: default_export readonly=1 tls=0
nbdkit: s3backer[1]: debug: s3backer: open readonly=1 exportname="" tls=0
nbdkit: s3backer[1]: debug: s3backer: default_export readonly=1 tls=0
nbdkit: s3backer[1]: debug: s3backer: open returned handle 0x7f4a5dffb580
nbdkit: s3backer[1]: debug: exitlast: open returned handle 0x7f4a5dffb580
nbdkit: s3backer[1]: debug: s3backer: prepare readonly=1
nbdkit: s3backer[1]: debug: exitlast: prepare readonly=1
nbdkit: s3backer[1]: debug: exitlast: get_size
nbdkit: s3backer[1]: debug: s3backer: get_size
nbdkit: s3backer[1]: debug: exitlast: can_zero
nbdkit: s3backer[1]: debug: exitlast: can_fast_zero
nbdkit: s3backer[1]: debug: exitlast: can_trim
...

My NBD plugin also ends up logging stuff to stderr (not shown) in this situation - presumably because it is logging via nbdkit_vdebug() and that goes to the same place.

Problem 2: When nbdkit(1) is invoked without the --verbose flag, logging completely disappears. It should be going to syslog but never shows up in /var/log/messages (even with --log=syslog).

Example:

$ ./s3backer --nbd --debug --listBlocks --readOnly s3backer-demo /dev/nbd1
2022-07-08 12:57:04 INFO: s3backer: executing /usr/sbin/nbdkit with these parameters:
2022-07-08 12:57:04 INFO: s3backer:   [00] "/usr/sbin/nbdkit"
2022-07-08 12:57:04 INFO: s3backer:   [01] "--read-only"
2022-07-08 12:57:04 INFO: s3backer:   [02] "--log=syslog"
2022-07-08 12:57:04 INFO: s3backer:   [03] "--filter=exitlast"
2022-07-08 12:57:04 INFO: s3backer:   [04] "--unix"
2022-07-08 12:57:04 INFO: s3backer:   [05] "/var/run/s3backer-nbd/0000000000000006_0000000000000538"
2022-07-08 12:57:04 INFO: s3backer:   [06] "s3backer"
2022-07-08 12:57:04 INFO: s3backer:   [07] "s3b_debug=true"
2022-07-08 12:57:04 INFO: s3backer:   [08] "s3b_listBlocks=true"
2022-07-08 12:57:04 INFO: s3backer:   [09] "s3b_readOnly=true"
2022-07-08 12:57:04 INFO: s3backer:   [10] "bucket=s3backer-demo"
2022-07-08 12:57:04 INFO: s3backer: started /usr/sbin/nbdkit as process 29116

What am I doing wrong? Thanks.

@rwmjones
Copy link
Member

rwmjones commented Jul 8, 2022

FYI upstream issues are here: https://gitlab.com/nbdkit/nbdkit/-/issues

The issue is that debug messages and error messages are handled separately.

Debug messages are only generated when --verbose is present, and are always sent to stderr: https://gitlab.com/nbdkit/nbdkit/-/blob/master/server/debug.c It's not possible to change this, but you can redirect fd 2 of the nbdkit subprocess to somewhere else (maybe to the logger command?)

Error messages are generated for error conditions, and they go to stderr or the syslog depending on the --log setting, as described in the manual. https://gitlab.com/nbdkit/nbdkit/-/blob/master/server/log.c

So it's working as expected as far as I can see. I would suggest for the debug messages that you redirect fd 2 of the nbdkit process after forking.

@rwmjones rwmjones closed this as completed Jul 8, 2022
@archiecobbs
Copy link
Author

OK - thanks for the explanation and sorry about targeting the wrong project!

asomers pushed a commit to asomers/nbdkit that referenced this issue Jan 13, 2023
VDDK 7.0.0 and 7.0.3 crash with GLIBC_TUNABLES (malloc debugging) and
latest glibc.  This likely indicates a bug in VDDK, but as we cannot
fix it we need to disable the extra hygiene checks instead.  For
information the stack trace is below.

  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  39	  magic = (((uintptr_t) p >> 3) ^ ((uintptr_t) p >> 11)) & 0xFF;
  [Current thread is 1 (Thread 0x7fea8056b640 (LWP 3905172))]
  (gdb) bt
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  libguestfs#1  malloc_usable_size (mem=0x0) at malloc-debug.c:405
  libguestfs#2  0x00007fea7fd5199b in greedy_realloc0 (p=p@entry=0x7fea805690c0,
      need=need@entry=2, size=size@entry=16) at ../src/basic/alloc-util.c:87
  libguestfs#3  0x00007fea7fd51b13 in netlink_container_parse (m=m@entry=0x555d34e77f80,
      container=0x555d34e77f90, rta=0x555d34e821f8, rt_len=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-message.c:1078
  libguestfs#4  0x00007fea7fd557ec in sd_netlink_message_rewind (genl=<optimized out>,
      m=<optimized out>) at ../src/libsystemd/sd-netlink/netlink-message.c:1324
  libguestfs#5  socket_read_message (rtnl=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-socket.c:375
  libguestfs#6  sd_netlink_read (usec=0, ret=0x7fea805692a0, serial=<optimized out>,
      rtnl=<optimized out>) at ../src/libsystemd/sd-netlink/sd-netlink.c:700
  libguestfs#7  sd_netlink_call (rtnl=<optimized out>, message=<optimized out>,
      ret=0x7fea805692a0, usec=0)
      at ../src/libsystemd/sd-netlink/sd-netlink.c:743
  libguestfs#8  0x00007fea7fd5818e in local_addresses (af=2, ret=0x7fea80569378,
      ifindex=0, context=0x0) at ../src/shared/local-addresses.c:90
  libguestfs#9  0x00007fea7fd590b9 in _nss_myhostname_gethostbyname3_r (
      name=<optimized out>, af=<optimized out>, host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594, ttlp=0x0, canonp=0x0)
      at ../src/nss-myhostname/nss-myhostname.c:373
  libguestfs#10 0x00007fea7fd593cb in _nss_myhostname_gethostbyname_r (
      name=0x7fea80569601 "pick", host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594) at ../src/nss-myhostname/nss-myhostname.c:514
  libguestfs#11 0x00007fea864491cb in __gethostbyname_r (
      name=0x7fea80569601 "pick", resbuf=0x7fea805695a0,
      buffer=<optimized out>, buflen=1024, result=<optimized out>,
      h_errnop=0x7fea80569594) at ../nss/getXXbyYY_r.c:273
  libguestfs#12 0x00007fea85bdcca4 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#13 0x00007fea85bd2bb2 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#14 0x00007fea85bba567 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#15 0x00007fea85bbab75 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#16 0x00007fea85bc63bd in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#17 0x00007fea85bc42ec in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#18 0x00007fea85bbf026 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#19 0x00007fea85bc0038 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#20 0x00007fea85c2fb23 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#21 0x00007fea85c2f160 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #22 0x00007fea85c2f1ba in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #23 0x00007fea85d0c0b9 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #24 0x00007fea85d079cb in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #25 0x00007fea85cb2831 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #26 0x00007fea85ca4702 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #27 0x00007fea85c9f76f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #28 0x00007fea85c7b59f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #29 0x00007fea85c7c11d in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #30 0x00007fea85bb4475 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #31 0x00007fea85bb4717 in VixDiskLib_Open ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #32 0x00007fea867b0dbb in vddk_open (readonly=<optimized out>)
      at /home/rjones/d/nbdkit/plugins/vddk/vddk.c:717
  #33 0x0000555d2a83d922 in plugin_open (c=0x555d34e81d10, readonly=0,
      exportname=<optimized out>, is_tls=<optimized out>)
      at /home/rjones/d/nbdkit/server/plugins.c:353
  #34 0x0000555d2a83572d in backend_open (b=0x555d34668db0, readonly=0,
      exportname=0x555d34dae020 "", shared=0)
      at /home/rjones/d/nbdkit/server/backend.c:292
  #35 0x0000555d2a83b937 in protocol_common_open (
      exportsize=exportsize@entry=0x7fea8056a9e0,
      flags=flags@entry=0x555d34e7f604,
      exportname=exportname@entry=0x555d34e7eeb0 "")
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:83
  #36 0x0000555d2a83e140 in finish_newstyle_options (exportsize=0x7fea8056a9e0,
      exportname_in=<optimized out>, exportnamelen=<optimized out>)
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:284
  #37 0x0000555d2a837ee0 in negotiate_handshake_newstyle_options ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:572
  #38 protocol_handshake_newstyle ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:904
  #39 protocol_handshake ()
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:55
  #40 handle_single_connection (sockin=<optimized out>, sockout=<optimized out>)
      at /home/rjones/d/nbdkit/server/connections.c:171
  #41 0x0000555d2a83e4e0 in start_thread (datav=0x555d34d987d0)
      at /home/rjones/d/nbdkit/server/sockets.c:354
  #42 0x00007fea863b4677 in start_thread (arg=<optimized out>)
      at pthread_create.c:434
  #43 0x00007fea86439320 in clone3 ()
      at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
mmuman pushed a commit to mmuman/nbdkit that referenced this issue Mar 4, 2024
mmuman pushed a commit to mmuman/nbdkit that referenced this issue Mar 4, 2024
Fix the Rust peername unit tests on BE architectures

Closes libguestfs#18

See merge request nbdkit/nbdkit!41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants