Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handling pathological xz files #17

Closed
mhenriks opened this issue Jun 13, 2022 · 1 comment
Closed

Handling pathological xz files #17

mhenriks opened this issue Jun 13, 2022 · 1 comment

Comments

@mhenriks
Copy link

Hi, KubeVirt's Containerized Data Importer project uses nbdkit internally to "import" xz compressed disk images. I just want to make you aware of an issue that has come our way: kubevirt/containerized-data-importer#2288

There are two problematic URLs, but I mostly want to bring your attention to the second: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2.xz

It is a single stream/single block xz file. Since the goal of our project is maximum compatibility, and there is no way to realistically support such pathological cases with nbdkit xz filter as-is, we are researching alternatives.

Do you think it makes sense for xz filter to handle these pathological files? I guess you'd have to do something similar to the gzip filter where the entire file is downloaded and extracted.
@rwmjones
Copy link
Member

nbdkit is managed as a gitlab project now: https://gitlab.com/nbdkit/nbdkit

I answered the query on the other ticket.

asomers pushed a commit to asomers/nbdkit that referenced this issue Jan 13, 2023
@rwmjones
tests: vddk: Disable GLIBC_TUNABLES for VDDK tests
03406a7 
VDDK 7.0.0 and 7.0.3 crash with GLIBC_TUNABLES (malloc debugging) and
latest glibc.  This likely indicates a bug in VDDK, but as we cannot
fix it we need to disable the extra hygiene checks instead.  For
information the stack trace is below.

  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  39	  magic = (((uintptr_t) p >> 3) ^ ((uintptr_t) p >> 11)) & 0xFF;
  [Current thread is 1 (Thread 0x7fea8056b640 (LWP 3905172))]
  (gdb) bt
  #0  malloc_check_get_size (mem=0x0)
      at /usr/src/debug/glibc-2.34.9000-13.fc36.x86_64/malloc/malloc-check.c:39
  libguestfs#1  malloc_usable_size (mem=0x0) at malloc-debug.c:405
  libguestfs#2  0x00007fea7fd5199b in greedy_realloc0 (p=p@entry=0x7fea805690c0,
      need=need@entry=2, size=size@entry=16) at ../src/basic/alloc-util.c:87
  libguestfs#3  0x00007fea7fd51b13 in netlink_container_parse (m=m@entry=0x555d34e77f80,
      container=0x555d34e77f90, rta=0x555d34e821f8, rt_len=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-message.c:1078
  libguestfs#4  0x00007fea7fd557ec in sd_netlink_message_rewind (genl=<optimized out>,
      m=<optimized out>) at ../src/libsystemd/sd-netlink/netlink-message.c:1324
  libguestfs#5  socket_read_message (rtnl=<optimized out>)
      at ../src/libsystemd/sd-netlink/netlink-socket.c:375
  libguestfs#6  sd_netlink_read (usec=0, ret=0x7fea805692a0, serial=<optimized out>,
      rtnl=<optimized out>) at ../src/libsystemd/sd-netlink/sd-netlink.c:700
  libguestfs#7  sd_netlink_call (rtnl=<optimized out>, message=<optimized out>,
      ret=0x7fea805692a0, usec=0)
      at ../src/libsystemd/sd-netlink/sd-netlink.c:743
  libguestfs#8  0x00007fea7fd5818e in local_addresses (af=2, ret=0x7fea80569378,
      ifindex=0, context=0x0) at ../src/shared/local-addresses.c:90
  libguestfs#9  0x00007fea7fd590b9 in _nss_myhostname_gethostbyname3_r (
      name=<optimized out>, af=<optimized out>, host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594, ttlp=0x0, canonp=0x0)
      at ../src/nss-myhostname/nss-myhostname.c:373
  libguestfs#10 0x00007fea7fd593cb in _nss_myhostname_gethostbyname_r (
      name=0x7fea80569601 "pick", host=0x7fea805695a0,
      buffer=0x7fea80569750 "\177", buflen=1024, errnop=0x7fea8056b4b8,
      h_errnop=0x7fea80569594) at ../src/nss-myhostname/nss-myhostname.c:514
  libguestfs#11 0x00007fea864491cb in __gethostbyname_r (
      name=0x7fea80569601 "pick", resbuf=0x7fea805695a0,
      buffer=<optimized out>, buflen=1024, result=<optimized out>,
      h_errnop=0x7fea80569594) at ../nss/getXXbyYY_r.c:273
  libguestfs#12 0x00007fea85bdcca4 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#13 0x00007fea85bd2bb2 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#14 0x00007fea85bba567 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#15 0x00007fea85bbab75 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#16 0x00007fea85bc63bd in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#17 0x00007fea85bc42ec in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#18 0x00007fea85bbf026 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#19 0x00007fea85bc0038 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#20 0x00007fea85c2fb23 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  libguestfs#21 0x00007fea85c2f160 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #22 0x00007fea85c2f1ba in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #23 0x00007fea85d0c0b9 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #24 0x00007fea85d079cb in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #25 0x00007fea85cb2831 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #26 0x00007fea85ca4702 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #27 0x00007fea85c9f76f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #28 0x00007fea85c7b59f in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #29 0x00007fea85c7c11d in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #30 0x00007fea85bb4475 in ?? ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #31 0x00007fea85bb4717 in VixDiskLib_Open ()
     from vddk-7.0.3/vmware-vix-disklib-distrib/lib64/libvixDiskLib.so.7
  #32 0x00007fea867b0dbb in vddk_open (readonly=<optimized out>)
      at /home/rjones/d/nbdkit/plugins/vddk/vddk.c:717
  #33 0x0000555d2a83d922 in plugin_open (c=0x555d34e81d10, readonly=0,
      exportname=<optimized out>, is_tls=<optimized out>)
      at /home/rjones/d/nbdkit/server/plugins.c:353
  #34 0x0000555d2a83572d in backend_open (b=0x555d34668db0, readonly=0,
      exportname=0x555d34dae020 "", shared=0)
      at /home/rjones/d/nbdkit/server/backend.c:292
  #35 0x0000555d2a83b937 in protocol_common_open (
      exportsize=exportsize@entry=0x7fea8056a9e0,
      flags=flags@entry=0x555d34e7f604,
      exportname=exportname@entry=0x555d34e7eeb0 "")
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:83
  #36 0x0000555d2a83e140 in finish_newstyle_options (exportsize=0x7fea8056a9e0,
      exportname_in=<optimized out>, exportnamelen=<optimized out>)
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:284
  #37 0x0000555d2a837ee0 in negotiate_handshake_newstyle_options ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:572
  #38 protocol_handshake_newstyle ()
      at /home/rjones/d/nbdkit/server/protocol-handshake-newstyle.c:904
  #39 protocol_handshake ()
      at /home/rjones/d/nbdkit/server/protocol-handshake.c:55
  #40 handle_single_connection (sockin=<optimized out>, sockout=<optimized out>)
      at /home/rjones/d/nbdkit/server/connections.c:171
  #41 0x0000555d2a83e4e0 in start_thread (datav=0x555d34d987d0)
      at /home/rjones/d/nbdkit/server/sockets.c:354
  #42 0x00007fea863b4677 in start_thread (arg=<optimized out>)
      at pthread_create.c:434
  #43 0x00007fea86439320 in clone3 ()
      at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rwmjones @mhenriks