CI test run does not reflect real production systems

[stefanb2]

No wonder that I can't even get passing tests on Ubuntu 14.04. The CI test has a hard-coded

ln -sf /bin/bash /bin/sh

which normal users aren't allowed on production systems. It's a bad idea anyway, because it would corrupt the Ubuntu installation.

IMHO the test cases and/or test runner needs to be updated to use /bin/bash everywhere.

[stefanb2]

test suite seems to be really unstable. Trying all cases (ruby runtest.rb -c, -c -n, -c -n -a) on Ubuntu 14.04, Fedora 23, Fedora 23 with make 3.81 in $PATH leaves me with 1 to 8 unexpected passes/failures.

I wish I had at least one combination with 0 :-/

[shinh]

Now runtest.rb sets SHELL=/bin/bash and "ln -sf /bin/bash /bin/sh" was removed from the travis CI setting: fc24bd2 I also made a bunch of changes for GNU make 4. Hopefully these changes mitigate your issue?

[stefanb2]

Your latest changes improve the situation somewhat. "ruby runtest.rb -c" now passes on my Ubunut 14.04 machine.

But other tests can still fail, I think there is something wrong with handling of echo handling when using ninja?

err_error_in_recipe.mk: FAIL
--- out.make    2016-05-03 09:56:37.074432172 +0300
+++ out.kati    2016-05-03 09:56:37.074432172 +0300
@@ -1,4 +1,5 @@
 === test ===
+/bin/bash -c "echo -e \"Makefile:2: *** foo.\" 2>&1 && false"
 Makefile:2: *** foo.

 === FILES ===
fail_ignore_error.mk: FAIL
--- out.make    2016-05-03 09:56:38.302433053 +0300
+++ out.kati    2016-05-03 09:56:38.302433053 +0300
@@ -1,5 +1,6 @@
 === test ===
 *** [test] Error 1
+/bin/bash -c "(false ) && (false ; true ) && (echo FAIL )"

 === FILES ===

fail_subshell_in_recipe.mk: FAIL
--- out.make    2016-05-03 09:56:38.330433073 +0300
+++ out.kati    2016-05-03 09:56:38.330433073 +0300
@@ -1,5 +1,6 @@
 === test ===
 *** [test] Error 1
+/bin/bash -c "(false ) && ((true) ; echo FAIL )"

 === FILES ===

[shinh]

Could you tell me the result of

$ cd out/err_error_in_recipe.mk && ninja

For me, the output is

[1/1] build test
FAILED: /bin/bash -c "echo -e "Makefile:2: *** foo." 2>&1 && false"
Makefile:2: *** foo.
ninja: build stopped: subcommand failed.

The line with "FAILED:" should be removed by https://github.com/google/kati/blob/master/runtest.rb#L114 . But it looks like your ninja doesn't have the FAILED: prefix?

[stefanb2]

I have two lines, where you have one line:

[1/1] build test
FAILED: test
/bin/bash -c "echo -e \"Makefile:2: *** foo.\" 2>&1 && false"
Makefile:2: *** foo.
ninja: build stopped: subcommand failed.

Maybe you are not using upstream master of ninja?

[shinh]

Ah, it seems very recent ninja outputs two lines. The above change works?

[stefanb2]

Yes, on Ubuntu 14.04 (make 3.81 in $PATH) and Fedora 23 (make 4.0 in $PATH) I now get passing result for -c, -c -n and -c -n -a

Closing this issue.