t_run_record_null test fails under Gentoo's sandbox

[mattst88]

3/4 umockdev / umockdev-run FAIL             0.43s   killed by signal 6 SIGABRT
19:02:26 MALLOC_PERTURB_=14 MALLOC_CHECK_=3 TOP_SRCDIR=/var/tmp/portage/dev-util/umockdev-0.17.9/work/umockdev-0.17.9 GI_TYPELIB_PATH=/var/tmp/portage/dev-util/umockdev-0.17.9/work/umockdev-0.17.9-abi_x86_64.amd64 LD_LIBRARY_PATH=/var/tmp/portage/dev-util/umockdev-0.17.9/work/umockdev-0.17.9-abi_x86_64.amd64 G_DEBUG=fatal-warnings,fatal-criticals,gc-friendly PATH=/var/tmp/portage/dev-util/umockdev-0.17.9/work/umockdev-0.17.9-abi_x86_64.amd64:/var/tmp/portage/dev-util/umockdev-0.17.9/temp/python3.10/bin:/usr/lib/portage/python3.9/ebuild-helpers/xattr:/usr/lib/portage/python3.9/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/13/bin /var/tmp/portage/dev-util/umockdev-0.17.9/work/umockdev-0.17.9/src/umockdev-wrapper /var/tmp/portage/dev-util/umockdev-0.17.9/work/umockdev-0.17.9-abi_x86_64.amd64/test-umockdev-run
----------------------------------- output -----------------------------------
stdout:
# random seed: R02Sd9b4e9f8747704a57550e0c81fcce32b
1..16
# Start of umockdev-run tests
ok 1 /umockdev-run/exit_code
ok 2 /umockdev-run/version
ok 3 /umockdev-run/pipes
ok 4 /umockdev-run/udevadm-block
ok 5 /umockdev-run/invalid-args
ok 6 /umockdev-run/invalid-device
ok 7 /umockdev-run/invalid-ioctl
ok 8 /umockdev-run/invalid-script
ok 9 /umockdev-run/invalid-program
Bail out! ERROR:test-umockdev-run.p/tests/test-umockdev-run.c:425:check_program_out: assertion failed (_tmp7_ == expected_out): ("" == "/dev/null character special file 1 3\n")
stderr:
**
ERROR:test-umockdev-run.p/tests/test-umockdev-run.c:425:check_program_out: assertion failed (_tmp7_ == expected_out): ("" == "/dev/null character special file 1 3\n")

Bisected to b8ccdf8 which adds the test.

I believe sandbox is preventing access to something like /tmp/umockdev.HBCSL1/dev/null? I haven't been able to identify precisely what access is causing the failure.

[martinpitt]

Thanks for the report! I don't have Gentoo in CI, but would be happy to add it. Do you have some steps how to do a Gentoo build in a container? https://hub.docker.com/search?q=gentoo is a bit non-obvious (and there are zero official or trusted publisher images).

[martinpitt]

Can you try to apply commit 1541a35 ? That should also help with investigating this failure.

[mattst88]

Thanks. With that commit applied I see basically the same thing:

ERROR:../tests/test-umockdev-run.vala:106:check_program_out: assertion failed (_tmp8_ == expected_out): ("" == "/dev/null character special file 1 3\n")
Bail out! ERROR:../tests/test-umockdev-run.vala:106:check_program_out: assertion failed (_tmp8_ == expected_out): ("" == "/dev/null character special file 1 3\n")

I'll ask around about Docker images and get back to you.

[martinpitt]

Interesting, that means that the stat call fails, but doesn't print any error message? 😢

[mattst88]

Dunno. Seems to work when I run it manually:

% sandbox stat -c  "%n %F %t %T" /dev/null
/dev/null character special file 1 3

[mattst88]

Docker files are available here: https://gitweb.gentoo.org/proj/docker-images.git

They are the ones used to produce the Docker images here: https://hub.docker.com/u/gentoo/

[martinpitt]

Right, I found these already, I just didn't know what "stage3" or "portage" or "nomultilib" etc. are. podman run -it --rm docker.io/gentoo/stage3 looks reasonable, but what do I do now? I.e. how do I install build dependencies, which ebuild file do I use (and from where), and which steps I run to get the ebuild? Sorry, I have zero knowledge about Gentoo (Debian/Ubuntu/Fedora guy here..)

I found https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-util/umockdev , but it seems there's a new .ebuild for every release; curling/parsing/sorting that feels clumsy, so I figure there's some higher level tooling around all that?

Thanks!

[mattst88]

Sorry for the delayed response.

Yes, there are higher level tools like emerge that handle dependencies, etc.

I don't have a lot of experience with Docker but I'd suggest something like

$ ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev --with-test-deps --onlydeps

which will install all necessary dependencies for the latest packaged version of umockdev. From there, you can do a normal build of umockdev and run tests from the git repo.

[martinpitt]

Hmm, seems this needs some initialization?

❱❱❱ podman run -it --rm --volume `pwd`:/source:ro docker.io/gentoo/stage3
88313f6fe768 / # ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev --with-test-deps --onlydeps
!!! Section 'gentoo' in repos.conf has location attribute set to nonexistent directory: '/var/db/repos/gentoo'
!!! Invalid Repository Location (not a dir): '/var/db/repos/gentoo'


!!! /etc/portage/make.profile is not a symlink and will probably prevent most merges.
!!! It should point into a profile within /var/db/repos/gentoo/profiles/
!!! (You can safely ignore this message when syncing. It's harmless.)


!!! Your current profile is invalid. If you have just changed your profile
!!! configuration, you should revert back to the previous configuration.
!!! Allowed actions are limited to --help, --info, --search, --sync, and
!!! --version.

# ls -l /etc/portage/make.profile
lrwxrwxrwx 1 root root 59 May  8 17:26 /etc/portage/make.profile -> ../../var/db/repos/gentoo/profiles/default/linux/amd64/17.1

# ls -l /var/db/repos/
total 0

# mkdir -p /var/db/repos/gentoo

# ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev --with-test-deps --onlydeps


!!! /etc/portage/make.profile is not a symlink and will probably prevent most merges.
!!! It should point into a profile within /var/db/repos/gentoo/profiles/
!!! (You can safely ignore this message when syncing. It's harmless.)


!!! Your current profile is invalid. If you have just changed your profile
!!! configuration, you should revert back to the previous configuration.
!!! Allowed actions are limited to --help, --info, --search, --sync, and
!!! --version.

I found https://wiki.gentoo.org/wiki/Portage_Multi_Stage_Dockerfile which gives some necessary initialization for /etc/portage/make.profile/make.defaults . But then

# ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev --with-test-deps --onlydeps
WARNING: One or more repositories have missing repo_name entries:

	/var/db/repos/gentoo/profiles/repo_name

NOTE: Each repo_name entry should be a plain text file containing a
unique name for the repository on the first line.


!!! It seems /run is not mounted. Process management may malfunction.

 * IMPORTANT: 13 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.

Calculating dependencies... done!

emerge: there are no ebuilds to satisfy "dev-util/umockdev".

emerge: searching for similar names...
emerge: Maybe you meant any of these: dev-util/meson, dev-util/re2c, dev-util/gperf?

[mattst88]

Ah, crap. Sorry. My inexperience with our Docker containers is showing. Looks like we need an emerge-webrsync before asking to install the dependencies.

Let me see if I can rope someone more knowledgeable than me in to help.

Thanks for your patience, your help, and your awesome software!

[mattst88]

https://github.com/gentoo/gentoo-docker-images#using-the-portage-container-as-a-data-volume might be more helpful than listening to me :)

[ajakk]

That README is unfortunately out of date, though it's close. I was able to get a working Gentoo environment with these two one-liners:

docker create -v /var/db/repos/gentoo --name myportagesnapshot gentoo/portage:latest /bin/true
docker run --interactive --tty --cap-add=CAP_SYS_ADMIN --cap-add=CAP_NET_ADMIN --cap-add=CAP_SYS_PTRACE --volumes-from myportagesnapshot gentoo/stage3:latest /bin/bash

[martinpitt]

Ah, thanks! So all that missing repo config comes from a second container image. At least with podman, a mere "create" is not enough, the container actually needs to run so that it copies its data into the volume. I didn't test this with docker (it doesn't work any more on Fedora). I also tidied this up a bit, to not leave the temporary portagesnapshot container around, and name the volume so that it can be cleaned up easily:

podman run  --rm -v gentooportage:/var/db/repos/gentoo docker.io/gentoo/portage:latest /bin/true
podman run -it --rm -v gentooportage:/var/db/repos/gentoo -v .:/source --cap-add=CAP_SYS_ADMIN --cap-add=CAP_NET_ADMIN --cap-add=CAP_SYS_PTRACE  docker.io/gentoo/stage3:latest /bin/bash

Inside that I can run

ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev --with-test-deps --onlydeps

Oh my, that does take a while.. All other umockdev tests take seconds up to ~ 3 minutes, here building the build deps alone takes more than 5 mins on my 8-core laptop (so probably more like 20 mins on the dual-core GitHub workflow workers).

After that,

/source$ VALAC=valac-0.56 meson /tmp/b
meson test -C /tmp/b

works fine:

1/7 umockdev / ioctl-tree                            OK               0.02s
2/7 umockdev:fails-valgrind / umockdev.py            OK               0.11s
3/7 umockdev / static-code                           OK               0.14s
4/7 umockdev:fails-valgrind / umockdev-vala          OK               0.77s
5/7 umockdev / umockdev-run                          OK               1.53s
6/7 umockdev:fails-valgrind / umockdev-record        OK               1.62s
7/7 umockdev / umockdev                              OK               3.30s


Ok:                 7   
Expected Fail:      0   
Fail:               0   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

So I suppose this is not the same as running this "sandbox". Is there some way to run the ebuild of your official portage against a git tree, instead of the hardcoded release version?

ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev also succeeds, but doesn't run the tests. Is there some way to tell it to build and run checks?

[mattst88]

Inside that I can run

ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev --with-test-deps --onlydeps

Oh my, that does take a while.. All other umockdev tests take seconds up to ~ 3 minutes, here building the build deps alone takes more than 5 mins on my 8-core laptop (so probably more like 20 mins on the dual-core GitHub workflow workers).

I'll check what all it needs to install and see if I can find some way of speeding this part up.

After that,

/source$ VALAC=valac-0.56 meson /tmp/b
meson test -C /tmp/b

works fine:

1/7 umockdev / ioctl-tree                            OK               0.02s
2/7 umockdev:fails-valgrind / umockdev.py            OK               0.11s
3/7 umockdev / static-code                           OK               0.14s
4/7 umockdev:fails-valgrind / umockdev-vala          OK               0.77s
5/7 umockdev / umockdev-run                          OK               1.53s
6/7 umockdev:fails-valgrind / umockdev-record        OK               1.62s
7/7 umockdev / umockdev                              OK               3.30s


Ok:                 7   
Expected Fail:      0   
Fail:               0   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

So I suppose this is not the same as running this "sandbox".

Right. I've reproduced the failure outside of emerge by first running the tests under emerge (see below) and then running the failing command under sandbox:

MALLOC_PERTURB_=14 \
PATH=/var/tmp/portage/dev-util/umockdev-0.17.10/work/umockdev-0.17.10-abi_x86_64.amd64:/var/tmp/portage/dev-util/umockdev-0.17.10/temp/python3.10/bin:/usr/lib/portage/python3.9/ebuild-helpers/xattr:/usr/lib/portage/python3.9/ebuild-helpers:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/13/bin \
TOP_SRCDIR=/var/tmp/portage/dev-util/umockdev-0.17.10/work/umockdev-0.17.10 \
LD_LIBRARY_PATH=/var/tmp/portage/dev-util/umockdev-0.17.10/work/umockdev-0.17.10-abi_x86_64.amd64 \
MALLOC_CHECK_=3 \
G_DEBUG=fatal-warnings,fatal-criticals,gc-friendly \
GI_TYPELIB_PATH=/var/tmp/portage/dev-util/umockdev-0.17.10/work/umockdev-0.17.10-abi_x86_64.amd64 \
	sandbox \
	/var/tmp/portage/dev-util/umockdev-0.17.10/work/umockdev-0.17.10/src/umockdev-wrapper \
	/var/tmp/portage/dev-util/umockdev-0.17.10/work/umockdev-0.17.10-abi_x86_64.amd64/test-umockdev-run

Is there some way to run the ebuild of your official portage against a git tree, instead of the hardcoded release version?

Yes. I'll add an ebuild that builds umockdev from git for this purpose.

ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev also succeeds, but doesn't run the tests. Is there some way to tell it to build and run checks?

Yes. FEATURES=test ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev will run the test suite after building.

I'll fire up podman and see if I can give some better advice after trying it out myself.

[martinpitt]

Many thanks @mattst88! With these instructions I was able to reproduce the bug, still with the hardcoded VALAC=valac-0.56. Then meson test -C /tmp/b works fine, and sandbox meson test -C /tmp/b fails in the way you reported. 🎉

I created an integration test, and the log shows the expected error on GitHub as well. It takes about 7 minutes, which is acceptable.

Next I'll try your umockdev-9999 ebuild, that's useful as well!

[martinpitt]

Next I'll try your umockdev-9999 ebuild

How would I do that, actually? I googled a bit, found https://wiki.gentoo.org/wiki/Ebuild#Live_ebuilds , and tried various syntaxes:

FEATURES=test ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev-9999
FEATURES=test ACCEPT_KEYWORDS="~*" emerge dev-util/umockdev=9999
FEATURES=test ACCEPT_KEYWORDS="~*" emerge '>=dev-util/umockdev-9999'

The latter two at least seem syntactically correct, but still fail with

• dev-util/umockdev-9999::gentoo (masked by: missing keyword)

But anyway, that's a minor detail -- I'm happy to use that later on. I'll look at fixing the actual test now.

SANDBOX_DEBUG=1 sandbox meson test --num-processes=1 -v umockdev-run does not show any violations, so it's not that. But emerging strace and stracing the whole thing shows that stat crashes:

22681 readlink("/tmp", 0x7ffd2876d1b0, 1023) = -1 EINVAL (Invalid argument)
22681 readlink("/tmp/umockdev.3LNRM1", 0x7ffd2876d1b0, 1023) = -1 EINVAL (Invalid argument)
22681 readlink("/tmp/umockdev.3LNRM1/dev", 0x7ffd2876d1b0, 1023) = -1 EINVAL (Invalid argument)
22681 readlink("/tmp/umockdev.3LNRM1/dev/null", "/dev/pts/1", 1023) = 10
22681 readlink("/dev", 0x7ffd2876d1b0, 1023) = -1 EINVAL (Invalid argument)
22681 readlink("/dev/pts", 0x7ffd2876d1b0, 1023) = -1 EINVAL (Invalid argument)
22681 readlink("/dev/pts/1", 0x7ffd2876d1b0, 1023) = -1 EINVAL (Invalid argument)
22681 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f78fc142ffb} ---
22681 +++ killed by SIGSEGV (core dumped) +++
22678 <... poll resumed>)               = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
22678 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_DUMPED, si_pid=22681, si_uid=0, si_status=SIGSEGV, si_utime=0, si_stime=0} ---

Thus somehow sandbox does not get along with umockdev's mocked device.

It's also not special about /dev/null, when I moved to /dev/full (1:7) it crashes the same way. I'm emerging gdb now, that'll take a while (are there any pre-built Gentoo packages for developers?)

[martinpitt]

I poked this for a while longer, and can't make sense of it. I'll skip that test under sandbox for the time being.

[mattst88]

Thanks a bunch! I don't blame you at all for punting on this. I couldn't make heads or tails of it either.