Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shunit2 has not been tested with the proper shell (and the current actual test has been failing) #121

Open
ko1nksm opened this issue Aug 20, 2019 · 10 comments
Assignees
Labels
Milestone

Comments

@ko1nksm
Copy link

@ko1nksm ko1nksm commented Aug 20, 2019

shunit2_misc_test.sh is tested with the shell that set in the environment variable $SHELL.

e.g.

# Note: the test script is prefixed with '#' chars so that shUnit2 does not
# incorrectly interpret the embedded functions as real functions.
testUnboundVariable() {
  unittestF="${SHUNIT_TMPDIR}/unittest"
  sed 's/^#//' >"${unittestF}" <<EOF
## Treat unset variables as an error when performing parameter expansion.
#set -u
#
#boom() { x=\$1; }  # This function goes boom if no parameters are passed!
#test_boom() {
#  assertEquals 1 1
#  boom  # No parameter given
#  assertEquals 0 \$?
#}
#SHUNIT_COLOR='none'
#. ${TH_SHUNIT}
EOF
  ( exec "${SHELL:-sh}" "${unittestF}" >"${stdoutF}" 2>"${stderrF}" )
  assertFalse 'expected a non-zero exit value' $?
  grep '^ASSERT:unknown failure' "${stdoutF}" >/dev/null
  assertTrue 'assert message was not generated' $?

However $SHELL is not the current running shell, but the shell when login.

test_runner tries to test all installed shells, but it test only login shell (probably bash only).

I did a quick hack to test, then the current (ba130d6 and v2.1.7) test failed.

(note: $SHUNIT_SHELL was used instead of $SHELL on v2.1.7)

$ git diff
diff --git a/test_runner b/test_runner
index a9871e3..60fde40 100755
--- a/test_runner
+++ b/test_runner
@@ -129,6 +129,7 @@ EOF

     # Check for existence of shell.
     shell_bin=${shell}
+    export SHELL=${shell} # or SHUNIT_SHELL=${shell} on v2.1.7
     shell_name=''
     shell_present=${RUNNER_FALSE}
     case ${shell} in
Result of ./test_runner
$ ./test_runner
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Linux 5.0.0-25-generic #26-Ubuntu SMP Thu Aug 1 12:04:58 UTC 2019 x86_64 x86_64

OS Name: Linux
OS Version: Ubuntu 19.04

### Test run info.
shells: /bin/sh ash /bin/bash /bin/dash /bin/ksh /bin/pdksh /bin/zsh
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/sh.
#
shell name: sh
shell version: unknown

--- Executing the 'shunit2_args' test suite. ---
non_default_test
test_non_default_ran

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 10 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=2)

--- Executing the 'shunit2_standalone' test suite. ---


#------------------------------------------------------------------------------
# Running the test suite with ash.
#
shell name: ash
shell version: 1.27.2

--- Executing the 'shunit2_args' test suite. ---
non_default_test
test_non_default_ran

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 10 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=2)

--- Executing the 'shunit2_standalone' test suite. ---


#------------------------------------------------------------------------------
# Running the test suite with /bin/bash.
#
shell name: bash
shell version: 5.0.3(1)-release

--- Executing the 'shunit2_args' test suite. ---
non_default_test
test_non_default_ran

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 10 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

OK

--- Executing the 'shunit2_standalone' test suite. ---


#------------------------------------------------------------------------------
# Running the test suite with /bin/dash.
#
shell name: dash
shell version: 0.5.10.2-4ubuntu1

--- Executing the 'shunit2_args' test suite. ---
non_default_test
test_non_default_ran

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 10 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=2)

--- Executing the 'shunit2_standalone' test suite. ---


#------------------------------------------------------------------------------
# Running the test suite with /bin/ksh.
#
shell name: ksh
shell version: AJM-93u+-2012-08-01-$

--- Executing the 'shunit2_args' test suite. ---
non_default_test
test_non_default_ran

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 10 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:expected a non-zero exit value
ASSERT:assert message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=6)

--- Executing the 'shunit2_standalone' test suite. ---


#------------------------------------------------------------------------------
# Running the test suite with /bin/pdksh.
#
runner:WARN unable to run tests with the pdksh shell


#------------------------------------------------------------------------------
# Running the test suite with /bin/zsh.
#
shell name: zsh
shell version: 5.5.1

--- Executing the 'shunit2_args' test suite. ---
non_default_test
test_non_default_ran

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 10 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNull was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=21)

--- Executing the 'shunit2_standalone' test suite. ---
@osamuaoki
Copy link
Contributor

@osamuaoki osamuaoki commented Oct 13, 2019

Actually, $SHELL is set by bash and some shells but not set by dash. So controlled build environment with cleaned environment variable under dash has undefined $SHELL.

This patch fixed error for testIssue84 for me for bash build-test started on dash. But dash still fails for dash build-test started on dash.

Key difference in set -x log is
BASH is lead by 3 +++

+++ assertTrue 'failure message for assertFalse was not generated' 0

DASH is lead by +

+ assertTrue failure message for assertFalse was not generated 1

Also right most value is different.

FULL LOG

BASH (with set -x):

testIssue84
+++ unittestF=/tmp/shunit.b5sPS5/tmp/unittest
+++ sed 's/^#//'
+++ exec /bin/bash /tmp/shunit.b5sPS5/tmp/unittest
+++ grep '^FAILED' /tmp/shunit.b5sPS5/tmp/stdout
+++ assertTrue 'failure message for assertFalse was not generated' 0
+++ eval '__shunit_lineno="";' if command '[' '"${1:-}"' = '"--lineno"' '];' then command '[' -n '"$2"' ']' '&&' '__shunit_lineno="[$2]' '";' shift '2;' fi
++++ __shunit_lineno=
++++ command '[' 'failure message for assertFalse was not generated' = --lineno ']'
++++ '[' 'failure message for assertFalse was not generated' = --lineno ']'
+++ command '[' 2 -lt 1 -o 2 -gt 2 ']'
+++ '[' 2 -lt 1 -o 2 -gt 2 ']'
+++ _shunit_shouldSkip
+++ command '[' 1 -eq 1 ']'
+++ '[' 1 -eq 1 ']'
+++ return 1
+++ shunit_message_=
+++ command '[' 2 -eq 2 ']'
+++ '[' 2 -eq 2 ']'
+++ shunit_message_='failure message for assertFalse was not generated'
+++ shift
+++ shunit_condition_=0
++++ expr 0 : '\([0-9]*\)'
+++ shunit_match_=0
+++ shunit_return=0
+++ command '[' -z 0 ']'
+++ '[' -z 0 ']'
+++ command '[' -n 0 -a 0 = 0 ']'
+++ '[' -n 0 -a 0 = 0 ']'
+++ command '[' 0 -ne 0 ']'
+++ '[' 0 -ne 0 ']'
+++ command '[' 0 -eq 0 ']'
+++ '[' 0 -eq 0 ']'
+++ _shunit_assertPass
++++ expr 18 + 1
+++ __shunit_assertsPassed=19
++++ expr 18 + 1
+++ __shunit_assertsTotal=19
+++ unset shunit_message_ shunit_condition_ shunit_match_
+++ return 0
+++ set +x

DASH:

testIssue84
+ unittestF=/tmp/shunit.y5DlXm/tmp/unittest
+ sed s/^#//
+ exec /bin/dash /tmp/shunit.y5DlXm/tmp/unittest
+ grep ^FAILED /tmp/shunit.y5DlXm/tmp/stdout
+ assertTrue failure message for assertFalse was not generated 1
+ eval __shunit_lineno=""; if command [ "${1:-}" = "--lineno" ]; then command [ -n "$2" ] && __shunit_lineno="[$2] "; shift 2; fi
+ __shunit_lineno=
+ command [ failure message for assertFalse was not generated = --lineno ]
+ command [ 2 -lt 1 -o 2 -gt 2 ]
+ _shunit_shouldSkip
+ command [ 1 -eq 1 ]
+ return 1
+ shunit_message_=
+ command [ 2 -eq 2 ]
+ shunit_message_=failure message for assertFalse was not generated
+ shift
+ shunit_condition_=1
+ expr 1 : \([0-9]*\)
+ shunit_match_=1
+ shunit_return=0
+ command [ -z 1 ]
+ command [ -n 1 -a 1 = 1 ]
+ command [ 1 -ne 0 ]
+ shunit_return=1
+ command [ 1 -eq 0 ]
+ _shunit_assertFail failure message for assertFalse was not generated
+ __shunit_testSuccess=1
+ _shunit_incFailedCount
+ expr 0 + 1
+ __shunit_assertsFailed=1
+ expr 18 + 1
+ __shunit_assertsTotal=19
+ [ 1 -gt 0 ]
+ echo \033[1;31mASSERT:\033[0mfailure message for assertFalse was not generated
ASSERT:failure message for assertFalse was not generated
+ unset shunit_message_ shunit_condition_ shunit_match_
+ return 1
+ set +x

@osamuaoki
Copy link
Contributor

@osamuaoki osamuaoki commented Oct 14, 2019

I have problem sourcing following script;

SH="/bin/dash"                                                                            
echo "SHELL='$SHELL'"
env -i SHELL=$SHELL bash ./test_runner -s${SH} -tshunit2_misc_test.sh
env -i SHELL=$SHELL dash ./test_runner -s${SH} -tshunit2_misc_test.sh
env -i SHELL=$SHELL  ksh ./test_runner -s${SH} -tshunit2_misc_test.sh
env -i SHELL=$SHELL mksh ./test_runner -s${SH} -tshunit2_misc_test.sh

By changing first line to use /bin/bash, this thing works like charm.

@osamuaoki
Copy link
Contributor

@osamuaoki osamuaoki commented Oct 14, 2019

Hmmm... even without patch, bash doesn't fail. even with patch, dash fails. Maybe I am getting confused.

I decided to test shunit2_misc_test.sh only for bash for Debian packaging for now. All others are tested for 5 shells: /bin/bash /bin/dash /bin/ksh /bin/mksh /bin/zsh

Is it OK to release 2.1.8pre to Debian unstable repository? Debian has 2.1.6 variant now.

Thanks for your nice package.

@osamuaoki
Copy link
Contributor

@osamuaoki osamuaoki commented Oct 17, 2019

Now I see what was happening.

The strange ${SHELL-sh} in shunit2_misc_test.sh is almost meant to use bash no matter what shell test_runner requests. That means cheating on testing code for each shell. That is not right thing to do.

When this is invoked from interactive bash shell $SHELL is already set as bash and bash is used.

But it it is started from other shell like:

 $ env -i dash ./test_runner -tshunit2_misc_test.sh

Things get bad nowadays.

This cheating worked mostly on old Ubuntu/Debian where /bin/sh was linked to bash. No more. /bin/sh is linked to dash now. So if you still want to cheat like this, ${SHELL-bash} needs to be here.

But that cheating fails on ksh since:

  • ksh --> $SHELL is /bin/ksh
  • mksh/dash/zsh --> No $SHELL set (So cheating works)
  • bash --> $SHELL is /bin/bash

So if we really have to cheat, use "bash" there.

I think this cheating seems to beat the purpose of having these test scripts.

dash:

...
TestIssue84
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
...
testColors
ASSERT:expected:<16> but was:<256>
shunit2:ERROR testColors() returned non-zero return code.
...

ksh:

shell name: ksh
shell version: AJM-93u+-2012-08-01-$

--- Executing the 'shunit2_misc' test suite. ---
./shunit2_misc_test.sh[315]: .: line 1282: PATH: parameter not set
./shunit2_misc_test.sh[970]: tearDown: not found [No such file or directory]
shunit2:WARN tearDown() returned non-zero return code.
./shunit2_misc_test.sh[973]: oneTimeTearDown: not found [No such file or directory]
shunit2:WARN oneTimeTearDown() returned non-zero return code.
ASSERT:unknown failure encountered running a test

Ran 0 tests.

FAILED (failures=1)

mksh, zsh -- looks same as dash since it uses dash by cheating under modern Debian.

(If I set SHELL to something, more errors come out from non-bash shells.)

I think we need to make test case for dash at least.

@williamdes
Copy link
Collaborator

@williamdes williamdes commented Dec 14, 2020

On Ubuntu

$ ./shunit2_misc_test.sh
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
Shell : /bin/bash
/bin/bash /tmp/shunit.RUnlAy/tmp/unittest /tmp/shunit.RUnlAy/tmp/stdout /tmp/shunit.RUnlAy/tmp/stderr
----------- stdoutf -----------
--- test_syntax_error

Ran 1 test.

FAILED (failures=1)
----------- stdoutfend -----------
----------- stderrF -----------
/tmp/shunit.RUnlAy/tmp/unittest: ligne 2: ${!#3442} : mauvaise substitution
shunit2:ERROR test_syntax_error() returned non-zero return code.
----------- stderrFend -----------
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

OK

On Alpine

/app/shunit2-2.1.8 # ./shunit2_misc_test.sh 
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
Shell : sh
sh /tmp/shunit.PdGOOP/tmp/unittest /tmp/shunit.PdGOOP/tmp/stdout /tmp/shunit.PdGOOP/tmp/stderr
----------- stdoutf -----------
--- test_syntax_error

Ran 1 test.

OK
----------- stdoutfend -----------
----------- stderrF -----------
/tmp/shunit.PdGOOP/tmp/unittest: eval: line 2: -334: not found
----------- stderrFend -----------
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=2)

SHELL="/bin/bash" ./test_runner made it work, thank you !
Ref: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/15811

## For the colors error

env -i dash ./test_runner -tshunit2_misc_test.sh did help me reproduce it

#------------------------------------------------------------------------------
# Running the test suite with /bin/zsh.
#
shell name: zsh
shell version: 5.8

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue69
testIssue77
testIssue84
Shell : sh
sh /tmp/shunit.OpClAb/tmp/unittest /tmp/shunit.OpClAb/tmp/stdout /tmp/shunit.OpClAb/tmp/stderr
----------- stdoutf -----------
--- test_syntax_error

Ran 1 test.

OK
----------- stdoutfend -----------
----------- stderrF -----------
/tmp/shunit.OpClAb/tmp/unittest: eval: line 2: -334: not found
----------- stderrFend -----------
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions
testIssue54
testColors
------- testColors ------------
got: 16
want: 16
------- testColorsend------------
------- testColors ------------
got: 
want: 256
------- testColorsend------------
ASSERT:expected:<> but was:<256>
shunit2:ERROR testColors() returned non-zero return code.
[: out of range
testColorsWitoutTERM

Ran 13 tests.

FAILED (failures=4)

@williamdes
Copy link
Collaborator

@williamdes williamdes commented Dec 16, 2020

@kward
Copy link
Owner

@kward kward commented Oct 24, 2021

Could you let me know if these are still broken on HEAD?

@kward kward self-assigned this Oct 24, 2021
@kward kward added the bug label Oct 24, 2021
@kward kward added this to the 2.1.9 milestone Oct 24, 2021
@ko1nksm
Copy link
Author

@ko1nksm ko1nksm commented Oct 25, 2021

It's still broken.

I am using zsh as my interactive shell on macOS. In other words, SHELL=/bin/zsh.
When I run ./test_runner, the test fails.

$ echo "$SHELL"
/bin/zsh

$ ./test_runner
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /bin/sh ash /bin/bash /bin/dash /bin/ksh /bin/mksh /bin/zsh
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/sh.
#
shell name: sh
shell version: 3.2.57(1)-release

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
ASSERT:skipping message was not generated
testStartSkippingWithoutMessage

Ran 3 tests.

FAILED (failures=1)
runner:WARN /bin/sh not passing

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue54
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=16)
runner:WARN /bin/sh not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK


#------------------------------------------------------------------------------
# Running the test suite with ash.
#
runner:WARN unable to run tests with the ash shell


#------------------------------------------------------------------------------
# Running the test suite with /bin/bash.
#
shell name: bash
shell version: 3.2.57(1)-release

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
ASSERT:skipping message was not generated
testStartSkippingWithoutMessage

Ran 3 tests.

FAILED (failures=1)
runner:WARN /bin/bash not passing

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue54
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=16)
runner:WARN /bin/bash not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK


#------------------------------------------------------------------------------
# Running the test suite with /bin/dash.
#
shell name: dash
shell version: unknown

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
ASSERT:skipping message was not generated
testStartSkippingWithoutMessage

Ran 3 tests.

FAILED (failures=1)
runner:WARN /bin/dash not passing

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue54
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=16,skipped=1)
runner:WARN /bin/dash not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK


#------------------------------------------------------------------------------
# Running the test suite with /bin/ksh.
#
shell name: ksh
shell version: Version AJM 93u+ 2012-08-01

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
ASSERT:skipping message was not generated
testStartSkippingWithoutMessage

Ran 3 tests.

FAILED (failures=1)
runner:WARN /bin/ksh not passing

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue54
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=16,skipped=1)
runner:WARN /bin/ksh not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK


#------------------------------------------------------------------------------
# Running the test suite with /bin/mksh.
#
runner:WARN unable to run tests with the mksh shell


#------------------------------------------------------------------------------
# Running the test suite with /bin/zsh.
#
shell name: zsh
shell version: 5.8

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
ASSERT:skipping message was not generated
testStartSkippingWithoutMessage

Ran 3 tests.

FAILED (failures=1)
runner:WARN /bin/zsh not passing

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue54
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=16,skipped=1)
runner:WARN /bin/zsh not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK

@ko1nksm
Copy link
Author

@ko1nksm ko1nksm commented Oct 25, 2021

Additional test results.

SHELL=/bin/sh ./test_runner -s /bin/sh [OK]
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /bin/sh
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/sh.
#
shell name: sh
shell version: 3.2.57(1)-release

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
testStartSkippingWithoutMessage

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue54
testIssue69
testIssue77
testIssue84
testIssue123
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

OK

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK
SHELL=/bin/bash ./test_runner -s /bin/bash [OK]
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /bin/bash
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/bash.
#
shell name: bash
shell version: 3.2.57(1)-release

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
testStartSkippingWithoutMessage

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue54
testIssue69
testIssue77
testIssue84
testIssue123
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

OK

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK
SHELL=/bin/dash ./test_runner -s /bin/dash [FAILED]
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /bin/dash
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/dash.
#
shell name: dash
shell version: unknown

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
testStartSkippingWithoutMessage

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue54
testIssue69
testIssue77
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=1,skipped=1)
runner:WARN /bin/dash not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK
SHELL=/bin/ksh ./test_runner -s /bin/ksh [FAILED]
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /bin/ksh
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/ksh.
#
shell name: ksh
shell version: Version AJM 93u+ 2012-08-01

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
testStartSkippingWithoutMessage

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:expected a non-zero exit value
ASSERT:assert message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
testIssue54
testIssue69
testIssue77
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=4,skipped=1)
runner:WARN /bin/ksh not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK
SHELL=/bin/zsh ./test_runner -s /bin/zsh [FAILED]
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /bin/zsh
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /bin/zsh.
#
shell name: zsh
shell version: 5.8

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
ASSERT:skipping message was not generated
testStartSkippingWithoutMessage

Ran 3 tests.

FAILED (failures=1)
runner:WARN /bin/zsh not passing

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
shunit2:ERROR testUnboundVariable() returned non-zero return code.
testIssue7
testIssue29
ASSERT:expected:<0> but was:<1>
testIssue54
testIssue69
ASSERT:failure message for assertEquals was not generated
ASSERT:failure message for assertNotEquals was not generated
ASSERT:failure message for assertNotNull was not generated
ASSERT:failure message for assertSame was not generated
ASSERT:failure message for assertNotSame was not generated
ASSERT:failure message for assertTrue was not generated
ASSERT:failure message for assertFalse was not generated
shunit2:ERROR testIssue69() returned non-zero return code.
testIssue77
ASSERT:failure of oneTimeSetUp() did not end test
ASSERT:failure of setUp() did not end test
ASSERT:failure of tearDown() did not end test
ASSERT:failure of oneTimeTearDown() did not end test
shunit2:ERROR testIssue77() returned non-zero return code.
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=16,skipped=1)
runner:WARN /bin/zsh not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.
OK
SHELL=/usr/local/bin/mksh ./test_runner -s /usr/local/bin/mksh [FAILED]
#------------------------------------------------------------------------------
# System data.
#

$ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64 x86_64 i386

OS Name: macOS Big Sur
OS Version: 11.6

### Test run info.
shells: /usr/local/bin/mksh
tests: shunit2_args_test.sh shunit2_asserts_test.sh shunit2_failures_test.sh shunit2_general_test.sh shunit2_macros_test.sh shunit2_misc_test.sh shunit2_shopt_test.sh shunit2_standalone_test.sh


#------------------------------------------------------------------------------
# Running the test suite with /usr/local/bin/mksh.
#
shell name: mksh
shell version: @(#)MIRBSD KSH R59 2020/10/31

--- Executing the 'shunit2_args' test suite. ---
custom_test
testCustomTestRan

Ran 2 tests.

OK

--- Executing the 'shunit2_asserts' test suite. ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertContains
testAssertNotContains
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse
testTooFewArguments
testTooManyArguments

Ran 12 tests.

OK

--- Executing the 'shunit2_failures' test suite. ---
testFail
testFailsWithArgs
testTooFewArguments
testTooManyArguments

Ran 4 tests.

OK

--- Executing the 'shunit2_general' test suite. ---
testSkipping
testStartSkippingWithMessage
testStartSkippingWithoutMessage

Ran 3 tests.

OK

--- Executing the 'shunit2_macros' test suite. ---
test:WARN ${LINENO} is not working for this shell. Tests will be skipped.
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK (skipped=24)

--- Executing the 'shunit2_misc' test suite. ---
testUnboundVariable
testIssue7
testIssue29
testIssue54
testIssue69
testIssue77
testIssue84
ASSERT:failure message was not generated
shunit2:ERROR testIssue84() returned non-zero return code.
testIssue123
shunit2:WARN [skipping] The ${BASH_SUBSHELL} variable is unavailable in this shell.
testPrepForSourcing
testExtractTestFunctions
testColors
testColorsWitoutTERM

Ran 12 tests.

FAILED (failures=1,skipped=1)
runner:WARN /usr/local/bin/mksh not passing

--- Executing the 'shunit2_shopt' test suite. ---
testNullglob

Ran 1 test.

OK (skipped=2)

--- Executing the 'shunit2_standalone' test suite. ---
testStandalone

Ran 1 test.

OK

cezanne added a commit to cezanne/shunit2 that referenced this issue Mar 1, 2022
Additional oneTimeTearDown() was called at shunit2 exit because there
seemed to be no set for __shunit_clean flag. Fix for kward#121
@williamdes
Copy link
Collaborator

@williamdes williamdes commented May 16, 2022

Can someone help me fixing this error ?
I am trying to package shunit2 into Debian (#155)

On my workstation tests pass but not on a CI: https://salsa.debian.org/debian/shunit2/-/jobs/2774040#L1015

shunit2:ERROR testIssue84() returned non-zero return code.

PS: cezanne@e736e3e does not work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants