testsuite fails with libusb 1.0.24 #115
Comments
|
While nixpkgs currently is on umockdev 0.13.1, I was able to also reproduce the failure with umockdev 0.15.2 - and moving libusb back to 1.0.23 did get the tests to succeed. |
Until martinpitt/umockdev#115 is resolved. Reported-In: NixOS#107420
|
I noticed something like this yesterday in a debian testing run. It also fails in an lsusb test now. Indeed Debian did the same version bump. So I need to re-do the recorded traces to also match this version. |
|
Unfortunately the test failure with Debian is slightly different -- lsusb just exits with 1 without printing a single error message, whereas in your case it did succeed (exit code 0) but did not print the expected device. But I'll start with investigating this on Debian (which is much easier for me), and with some luck it is the same root cause. I bisected this to libusb/libusb@c3864c8 which is unfortunately very big. To be continued later.. |
libusb changed the sysfs parser to require attributes be terminated with a newline. Adjust /umockdev-testbed-usb/lsusb accordingly. Fixes one half of issue #115 [1] libusb/libusb@c3864c822b
libusb 1.0.24 changed the sysfs parser to require attributes be terminated with a newline [1]. Update umockdev-run to keep trailing whitespace (in particular, newlines) unmodified, so that the mocked testbed reproduces them faithfully. Do the minimal changes to the pre-recorded devices to get along with the new libusb. Adjust /umockdev-testbed-usb/lsusb accordingly to terminate busnum/devnum with line feeds. Fixes #115 [1] libusb/libusb@c3864c822b
|
Hmm, this doesn't seem to do the trick for me: When just applying that patch on top of 0.15.2, I get the following output: When building |
|
@flokli : Interesting, that |
|
@flokli : Hm, I have tried for an hour how, stumbling around nix-env, building a default.nix that uses a local source checkout, etc., and I'm not really getting anywhere. Could you show me the commands to run in a container (docker works as well, of course) that will install the build deps and build/test umockdev? I'm happy to massage them into CI, but I need some help with grasping how to do things in nix. Thanks! |
|
You can build the package in one of the following ways:
|
|
Thanks @jtojnar! This looks workable, especially solution 3. Note for myself: Add I get a different failure, though: This seems related to commit 3cc547f: I attempted to fix that, but it's apparently not working just yet. I'll keep looking into this. @flokli : FYI, the |
|
The test-static-code failure is weird.. It gets called (unnecessarily, for technical reasons) through umockdev-wrapper, and it seems from the build tree the linker gets upset: It does seem to link against the right libc: I am at loss with this. The nix build env is rather hard to debug, as pretty much nothing is "in the system", which breaks drilling down to invididual build/test steps. |
|
Commit 0e60ddb adds a nix test, which succeeds. This hacks around the test-static-code failure by dropping the test from meson.build. Help with fixing this properly would be very much appreciated! But at least that will now keep the general build working, so 0.15.3 should be by and large ok for you? Thanks @flokli for the report and @jtojnar for your build recipe! |
|
Hmm, I am also getting the For CI, it might be better to use |
|
I could reproduce it in Docker by running the environment in $ docker exec -ti zen_gagarin env GI_TYPELIB_PATH='/tmp/nix-build-umockdev-0.15.2.drv-0/source/build:/nix/store/6j415gpqni7cj91xkk52dri9rg40gf21-gobject-introspection-1.66.1/lib/girepository-1.0:/nix/store/qh9f1728pb9ybp677vqbz39pv12z3cx4-libgudev-234/lib/girepository-1.0:/nix/store/6j415gpqni7cj91xkk52dri9rg40gf21-gobject-introspection-1.66.1/lib/girepository-1.0:/nix/store/qh9f1728pb9ybp677vqbz39pv12z3cx4-libgudev-234/lib/girepository-1.0:/nix/store/6j415gpqni7cj91xkk52dri9rg40gf21-gobject-introspection-1.66.1/lib/girepository-1.0:/nix/store/qh9f1728pb9ybp677vqbz39pv12z3cx4-libgudev-234/lib/girepository-1.0' PATH='/tmp/nix-build-umockdev-0.15.2.drv-0/source/build:/nix/store/9zmnxxd2lrfcvzyncsc8c9d0znkxd0wk-gobject-introspection-1.66.1-dev/bin:/nix/store/biv1ds2x8d5ji0yj7qxv44b40xid312g-glib-2.66.3-dev/bin:/nix/store/40aa067kyqc67dff1lzfcslivqx3ymkg-gettext-0.21/bin:/nix/store/lljddzamy684ah9pxzkr80x2qbc0fbg7-glib-2.66.3-bin/bin:/nix/store/w1xyc9wr5lvcakzz1n4q8mhs0v9xp7x5-gtk-doc-1.33.0/bin:/nix/store/crakvb0jk50ap9nczzvdlijds7rddhrk-meson-0.56.0/bin:/nix/store/09mmkj13p52yqlbsn77549qnr4hfjqbp-ninja-1.10.2/bin:/nix/store/d2bnc0h350yz0wlqbizhzb58lhmy1k6j-pkg-config-wrapper-0.29.2/bin:/nix/store/92gqx3fqkmzshikyp1mqx4ds23bz729s-vala-0.48.9/bin:/nix/store/m7080pw0ryjk0jhljp55rq1hd2qy8gki-python3-3.8.6/bin:/nix/store/20jz8krnn4g4iilf9fryzadjifs8pg17-which-2.21/bin:/nix/store/nx19q4zadh0m7j3jw4qq94spv197prfa-usbutils-012/bin:/nix/store/kg1ck8dgi10yqwmm20gi7gx5m3sr5jdq-patchelf-0.12/bin:/nix/store/b96dqbx6pri2xp2xxlq6i269virrdaw6-gcc-wrapper-9.3.0/bin:/nix/store/wmzmnnrj780268ybilmcfyd3grn5qzi6-gcc-9.3.0/bin:/nix/store/7niy0yd0ygv0wa05kl5l46x6fdflpwf1-glibc-2.32-10-bin/bin:/nix/store/8gsdifv2q3hnq42v7ddk38nmiagmjd6c-coreutils-8.32/bin:/nix/store/ip7xkk70anay3k7bp2cr6aqdg0mcdja2-binutils-wrapper-2.31.1/bin:/nix/store/kcp0y7g1ir4dxq2gqz687i4vb7gqy8s0-binutils-2.31.1/bin:/nix/store/7niy0yd0ygv0wa05kl5l46x6fdflpwf1-glibc-2.32-10-bin/bin:/nix/store/8gsdifv2q3hnq42v7ddk38nmiagmjd6c-coreutils-8.32/bin:/nix/store/biv1ds2x8d5ji0yj7qxv44b40xid312g-glib-2.66.3-dev/bin:/nix/store/40aa067kyqc67dff1lzfcslivqx3ymkg-gettext-0.21/bin:/nix/store/lljddzamy684ah9pxzkr80x2qbc0fbg7-glib-2.66.3-bin/bin:/nix/store/n572k0545dyc18vbzczqbrhh7h7rn4x1-systemd-247.2/bin:/nix/store/8gsdifv2q3hnq42v7ddk38nmiagmjd6c-coreutils-8.32/bin:/nix/store/khvalqldk47cx8d54gik9jrqsinzmwng-findutils-4.7.0/bin:/nix/store/6wzkn6gdpj3psqjgbfx858wd3dhanj12-diffutils-3.7/bin:/nix/store/q3ppqi1npx3vnsdlk8hk9s4p3adc4ygp-gnused-4.8/bin:/nix/store/134fdr0yrl4wgrrasxn9d3s8vzcm5lxq-gnugrep-3.6/bin:/nix/store/y9rbfriadb1gq5l8hjm8476cvjjq38ww-gawk-5.1.0/bin:/nix/store/gm4gjb3pkyxyj54kb4fizldvr60cpb2l-gnutar-1.32/bin:/nix/store/svhnwsc7rbzrfcfn0y1hsry1m8yi0y88-gzip-1.10/bin:/nix/store/pj616ldh4sbmqj7ad6dhmfcawfzgqpqf-bzip2-1.0.6.0.1-bin/bin:/nix/store/6mic0pcmp4va0ghx7hn5wl3jl4ll4h2q-gnumake-4.3/bin:/nix/store/4l7wsi6h6283194r6fqy1731qxlagq62-bash-4.4-p23/bin:/nix/store/wav2vx6iaxvkmzcwzwa127p6jri4rqdz-patch-2.7.6/bin:/nix/store/hzvfm203h5argiylhqprnpwznxxbdmj3-xz-5.2.5-bin/bin' MALLOC_CHECK_='3' G_DEBUG='fatal-warnings,fatal-criticals,gc-friendly' LD_LIBRARY_PATH='/tmp/nix-build-umockdev-0.15.2.drv-0/source/build' TOP_SRCDIR='/tmp/nix-build-umockdev-0.15.2.drv-0/source' /tmp/nix-build-umockdev-0.15.2.drv-0/source/src/umockdev-wrapper /tmp/nix-build-umockdev-0.15.2.drv-0/source/tests/test-static-code
Error relocating /tmp/nix-build-umockdev-0.15.2.drv-0/source/build/libumockdev-preload.so.0: __snprintf_chk: symbol not found
Error relocating /tmp/nix-build-umockdev-0.15.2.drv-0/source/build/libumockdev-preload.so.0: __memcpy_chk: symbol not found
Error relocating /tmp/nix-build-umockdev-0.15.2.drv-0/source/build/libumockdev-preload.so.0: __fprintf_chk: symbol not foundIt looks like it does not actually seem to be using the correct $ docker exec -ti zen_gagarin ldd /tmp/nix-build-umockdev-0.15.2.drv-0/source/build/libumockdev-preload.so.0
/lib/ld-musl-x86_64.so.1 (0x7fad1b57c000)
libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fad1b57c000)
libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fad1b57c000)
$ docker exec -ti zen_gagarin nix-shell -I nixpkgs=channel:nixos-unstable -p elfutils --run "readelf -d /tmp/nix-build-umockdev-0.15.2.drv-0/source/build/libumockdev-preload.so.0"
Dynamic section at offset 0x10b30 contains 30 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [libumockdev-preload.so.0]
0x000000000000001d (RUNPATH) Library runpath: [/nix/store/9k8il9x354dx6mvqrj0s7ndi7pz5pc8z-umockdev-0.15.2/lib64:/nix/store/9k8il9x354dx6mvqrj0s7ndi7pz5pc8z-umockdev-0.15.2/lib:/nix/store/1yvpgm763b3hvg8q4fzpzmflr5674x4j-glibc-2.32-10/lib:/nix/store/isy60my0ijjzh49rscgdb1i2457nf7lp-gcc-9.3.0-lib/lib]
0x000000000000000c (INIT) 0x4000
0x000000000000000d (FINI) 0xbef4
0x0000000000000019 (INIT_ARRAY) 0x11b18
0x000000000000001b (INIT_ARRAYSZ) 16 (bytes)
0x000000000000001a (FINI_ARRAY) 0x11b28
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x0000000000000004 (HASH) 0x200
0x000000006ffffef5 (GNU_HASH) 0x590
0x0000000000000005 (STRTAB) 0x1408
0x0000000000000006 (SYMTAB) 0x808
0x000000000000000a (STRSZ) 1684 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000003 (PLTGOT) 0x11d50
0x0000000000000002 (PLTRELSZ) 1632 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x2c80
0x0000000000000007 (RELA) 0x1c30
0x0000000000000008 (RELASZ) 4176 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000000000001e (FLAGS) BIND_NOW
0x000000006ffffffb (FLAGS_1) Flags: NOW
0x000000006ffffffe (VERNEED) 0x1ba0
0x000000006fffffff (VERNEEDNUM) 2
0x000000006ffffff0 (VERSYM) 0x1a9c
0x000000006ffffff9 (RELACOUNT) 158
0x0000000000000000 (NULL) 0x0 |
|
The If this targets github actions, it's probably easier to use https://github.com/marketplace/actions/install-nix (which should have the sandbox enabled). |
|
"musl leaking in" does sound like a good hint.. It would explain both the umockdev does work with musl in general, I fixed it up not too long ago for Alpine (which also runs in CI). But apparently it somehow gets linked to glibc and musl at the same time? @flokli: I do run the tests in GitHub actions right now, but I don't want to use that install-nix actions -- it would (1) mean that it's not possible to run and investigate the test locally, and (2) make it hard to move CI someplace else (I just moved it from Travis). In other words, I only support OSes that can be run in a container. That can be a privileged podman or docker container, so if there is any way to enable the sandboxing with these, I'm all ears. |
|
@jtojnar : I tried to move to in other words, this is still the old autotools build system, and https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/libraries/umockdev/default.nix is indeed much older than https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/umockdev/default.nix . So it seems the other way around, master is newer than unstable, and thus a better CI target? |
Well, the Nix builds outside of Docker are sandboxed so the musl cannot be an issue there.
Yeah, that's what I meant by it "does not yet contain the switch to meson based builds". The issue with using master is that it might not have prebuilt binaries available yet, depending on how busy our infrastructure is.
See https://github.com/NixOS/docker#limitations for the necessary configuration. |
Yes, the sandbox can be enabled, if you can start a privileged containers: https://github.com/NixOS/docker#limitations I proposed using the install nix action because then Nix is installed on the runner directly (for that run) and you don't need to mess with wrapping it in docker/podman in CI at all. Locally reproducing can be done on systems with Nix installed, or doing the privileged container dance there. |
|
A-ha! I enabled sandboxing; locally for a developer, running an user podman container with The exact same errors happen in GitHub CI. Does that faithfully reproduce what you are seeing? Note, I kept this change in a "nixos" branch for now, as I don't want to land changes in master which break the tests. |
|
How do you debug failures like this? I. e. after a failing obviously does not work, and NixOS/nix#1147 and nix-shell sound promising, but I can't put it together.. |
|
The Also, I am experiencing just the |
You might be able to use I never tried this through podman, but it might work :-) |
|
Thanks! I tried to replace |
|
@martinpitt This probably has exceeded the amount of characters I'd feel comfortable with having inline, but if you create a let
pkgs = (import (builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs/archive/master.tar.gz"; }) {});
in pkgs.umockdev.overrideAttrs (attrs: {
src = /source;
patches = [];
preCheck = "";
doCheck = true;
nativeBuildInputs = attrs.nativeBuildInputs ++ [ pkgs.breakpointHook ];
})and run Running |
|
|
|
I welcome having a default.nix file, it's much clearer than cramming it into a single command indeed. The version above fails with I'm googling around for similar recipes (it's hard to find one), as to me the syntax is not quite intuitive. But as this introduces a It lands in the wrong directory, though, so that all values in I.e. it seems it should really chroot into the /var/lib/cntr/ subdirectory (there is even a bin/sh), but I don't see an option for this. Nor does |
|
IIRC, |
|
Hm, I still don't understand this -- there must be some easy way to get a shell in the very build environment that If |
I did try that, but it is rather difficult with all the |
|
Sorry for this being so frustrating. Probably the easiest way is to either use the Nix installer on most other distributions, or one of the NixOS images shipped on the website: https://nixos.org/download.html |
|
I started a fedora 33 VM, and ran (as user): curl -L https://nixos.org/nix/install | sh
. ~/.nix-profile/etc/profile.d/nix.sh
nix-env -i git
nix-env -i cntr
git clone -b nixos https://github.com/martinpitt/umockdev.git
cat <<EOG > ~/default.nix
let pkgs = (import (builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs/archive/master.tar.gz"; }) {});
in pkgs.umockdev.overrideAttrs (attrs: {
src = /home/admin/umockdev;
patches = [];
preCheck = "";
doCheck = true;
nativeBuildInputs = attrs.nativeBuildInputs ++ [ ];
})
EOG
nix-build ~/default.nixThis reproduces exactly the failures that I see in the container. When I add I can't run it through sudo either, as all of these commands are specific to my I also tried to run the whole nix setup as user root, but that fails also: So I'm afraid I'm still none the wiser -- I really need some way to drill down into the test, in an interactive shell, with iterating meson, strace, gdb, etc. It seems that the behaviour in a VM is not any different to a container. |
In that environment, `lsusb` errors with "Couldn't open device, some information will be missing" and can't figure out the bus/device name (they are shown as 0). Make the test more lax in a Nix build for now, until this gets debugged properly (which is really hard, see #115).
|
I prodded around a bit by hacking the source (
Debugging this "blindly" is just too cumbersome, so I hacked the test to be a bit more lenient on NixOS for the time being. At least all the other tests can run then, and we prevent further regressions. That leaves the test-umockdev-vala failure. That fails exactly here, i.e. relative paths into the trapped /sys are not working. Everything before (absolute paths to trapped /sys) works fine. This is relatively new, from commit 946f3dd. The reason is again that the NixOS sandbox does not actually have With these changes, the nix tests succeed, so I landed that branch on master. @flokli, can you please check if current master now works alright for you? If so, I'm happy to publish a new release, and then you can clean up the I'd still like to figure out what is going wrong in |
|
@martinpitt wow, thanks for being so persistent despite the bad debuggability. I'll try to build it on that commit and will report back.
There's probably not much documentation about it, but Nix does set up some seccomp filters here and the rest of the sandbox environment setup, including filesystem mounts happens here. |
|
I confirmed I'm able to build and run tests in 71014c2 without any patches on top, in the sandbox, see NixOS/nixpkgs#108317 for the PR. |
|
@flokli: That lsusb test case tests simulated devices. It could still be some weird side effect of the machine not having a real /sys , but that's a bit hard to debug. That nixpkgs PR looks pleasantly green, so that seems good enough? I'll make a 0.15.4 then. Thanks for your help! |
While doing a nixpkgs bump, I tried compiling umockdev, and realized tests in umockdev suddenly broke:
I bisected this to the "libusb 1.0.23 -> 1.0.24" bump in nixpkgs (commit id 24342209d4011e4ca2d68c1507403d7f3cdd5607 there)
Can you reproduce this on your systems, with libusb 1.0.24?
The text was updated successfully, but these errors were encountered: