make check fails tests on Alpine Linux

[clandmeter]

Building on Alpine Linux (musl libc/busybox based distro) is successful but make check fails:

http://tpaste.us/JMml

ps I had to add diffutils as busybox diff seems not suported.

[brarcher]

From the logs you have posted it appears that the checkmk part of Check is failing its unit tests. That is a framework that helps generate test cases, and uses awk to do it. My guess is that the awk being used does not support all the features that checkmk awk script expects.

Are you expecting to use checkmk as well as libcheck? What awk is being used?

[clandmeter]

We are using awk from busybox.
I added gawk to checkdepends and now it starts tests but stalls at:

PASS: test/check_checkmk
============================================================================
Testsuite summary for Check 0.12.0
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/home/clandmeter/repositories/aports/main/check/src/check-0.12.0/checkmk'
make[2]: Leaving directory '/home/clandmeter/repositories/aports/main/check/src/check-0.12.0/checkmk'
make[1]: Leaving directory '/home/clandmeter/repositories/aports/main/check/src/check-0.12.0/checkmk'
Making check in tests
make[1]: Entering directory '/home/clandmeter/repositories/aports/main/check/src/check-0.12.0/tests'
make  check-TESTS
make[2]: Entering directory '/home/clandmeter/repositories/aports/main/check/src/check-0.12.0/tests'
make[3]: Entering directory '/home/clandmeter/repositories/aports/main/check/src/check-0.12.0/tests'
PASS: test_check_nofork_teardown.sh
PASS: test_check_nofork.sh
PASS: test_set_max_msg_size.sh
PASS: test_tap_output.sh
PASS: test_xml_output.sh
PASS: test_log_output.sh
PASS: test_output.sh

how long are these tests suppose to take?

[brarcher]

Check supports test timeouts. There a several tests which verify that the test timeout functionality works as expected. The check_check and check_check_export tests can take 10min or so to run because of the timeout checking.

[clandmeter]

Strange, the tests keep running even after 30 minutes. Looks like something is wrong.
Any idea how to debug?

[brarcher]

To debug, find the check_check test executable in the tests directory and execute it. It should output logs as it goes. Can you post the logs after running for 10 min or so?

…

On October 24, 2017 4:41:29 AM EDT, Carlo Landmeter ***@***.***> wrote: Strange, the tests keep running even after 30 minutes. Looks like something is wrong. Any idea how to debug? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #138 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[clandmeter]

./check_check
Running suite(s): Fork Sub
100%: Checks: 3, Failures: 0, Errors: 0
check_check_fork.c:38:P:Core:test_inc:0: Passed
check_check_fork.c:47:P:Core:test_nofork_sideeffects:0: Passed
check_check_fork.c:54:P:Core:test_nofork_pid:0: Passed
Running suite(s): Fix Sub
0%: Checks: 1, Failures: 1, Errors: 0
check_check_fixture.c:36:S:Fix Sub:unchecked_setup:0: Test failure in fixture
Running suite(s): Check Servant

[clandmeter]

I created a strace for you: http://sprunge.us/SaIj

[brarcher]

I appreciate the help. Unfortunately, the strace only shows the parent process and not the forked children. If you are able, can you run with CK_FORK=no (so that fork is not used) and post that strace? I'll also try to download Alpine Linux for a VM and reproduce the issue.

[clandmeter]

here you go http://sprunge.us/SSUX

[brarcher]

Thanks for posting the logs. Unfortunately, I'm not able to determine much from them.

I'm attempting to reproduce the issue myself so I can further debug it. In doing so, I'm running into some difficulty in setting up an Alpine VM as I'm unfamiliar with some of the tools its uses. I'll try the docker image instead and see if I can get further.

[brarcher]

I was able to use the Docker image of Alpine 3.6, install the necessary package, and install Check and run its tests.

/workdir/check/check/tests # time make check
make  check-TESTS
make[1]: Entering directory '/workdir/check/check/tests'
make[2]: Entering directory '/workdir/check/check/tests'
PASS: check_check_export
PASS: check_check
PASS: test_output.sh
PASS: test_check_nofork.sh
PASS: test_check_nofork_teardown.sh
PASS: test_xml_output.sh
PASS: test_log_output.sh
PASS: test_set_max_msg_size.sh
PASS: test_tap_output.sh
============================================================================
Testsuite summary for Check 0.12.0
============================================================================
# TOTAL: 9
# PASS:  9
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[2]: Leaving directory '/workdir/check/check/tests'
make[1]: Leaving directory '/workdir/check/check/tests'
real	5m 51.47s
user	0m 0.13s
sys	0m 0.32s

On a non-docker image are you seeing the tests hang indefinitely? Note that the check_check and check_check_export tests have output which looks like it is failing. However, those tests launch subtest whose job it is to demonstrate that assertions do cause tests to fail. If the subtests fail when they are expected, the overall test programs pass.

[clandmeter]

I think i found the issue. We are using hardened kernel with deter_bruteforce set and is triggered after excessive forking. If I disable this option the tests will succeed. The problem is we cannot disable this grsec options on our builders so if we want to run tests we need to work around it.

Any idea?

[brarcher]

Hm. If the kernel is preventing excessive forking, then users of libcheck may have issues running their tests on Alpine linux, as each test is forked into its own memory space.

If you want Check to work on Apline and not use fork, it can be configured with --disable-fork. Using this you should see the following output from configure near the end:

==========================================
Summary of Check 0.12.0 options:

fork mode ............................ no
...

Forking tests is a pretty nice feature, as it helps isolate tests from one another. However, it sounds like a reasonable trade-off if the forking behavior is not allowed.

[brarcher]

Oh, I misunderstood. The issue is only with the builders. Either way, the unit tests by default will use fork. The only way to run the tests without using fork is to disallow fork for the resulting library, which affects the features provided to the end user.

If you want a spot check, you could first compile and test without fork, and if that succeeds configure again with fork and only build. You will then run some unit tests, though skip the unit tests which require fork.