Skip to content
Permalink
Browse files

tests: enable shellcheck for shell-based plugins and flake8 for pytho…

…n-based plugins

Many plugins are still failing these tests.
These expected failures are listed in a file
(t/test-exception-wrapper.expected-failures)
A wrapper script is used for running the tests and comparing the result
with the expectation (based on the file being listed in the above file).

This allows to test all new plugins, while ignoring all known failures.
  • Loading branch information...
sumpfralle committed Aug 12, 2019
1 parent 5386013 commit e5abdeeb1269c4c3df0211c56f7d52c7704c5696
Showing with 599 additions and 21 deletions.
  1. +2 −0 .travis.yml
  2. +55 −0 t/test-exception-wrapper
  3. +492 −0 t/test-exception-wrapper.expected-failures
  4. +50 −21 t/test.t
@@ -5,7 +5,9 @@ addons:
packages:
- devscripts
- python
- python-flake8
- python3
- python3-flake8
- ruby
- php-cli
- gawk
@@ -0,0 +1,55 @@
#!/bin/sh
#
# Run a check for a given file. A failure is tolerated (and expected), if the filename is listed
# in a file containing expected failures.
#
# Parameters: SHELL_DIALECT SCRIPT_FILENAME
#
# See EXPECTED_FAILURES_LIST_FILENAME for the filename pattern of a file, containing the relative
# names of all scripts, that are expected to fail. This wrapper script will fail, if the exit
# status of the test does not match the expectated result (i.e. it fails but should pass or it
# passes while being listed in the EXPECTED_FAILURES_LIST_FILENAME file).
#

set -eu

[ $# -lt 2 ] && echo >&2 "Insufficient number of arguments: expecting SCRIPT_FILENAME and one or more COMMAND tokens" && exit 1


SCRIPT_FILENAME="$1"
shift
EXPECTED_FAILURES_LIST_FILENAME="$0.expected-failures"

REPOSITORY_DIR=$(cd "$(dirname "$0")/.." && pwd)


[ ! -e "$SCRIPT_FILENAME" ] && echo >&2 "Failed to find script: $SCRIPT_FILENAME" && exit 3


# determine, whether the script is mentioned in the exclusion file
relative_script_filename=$(realpath --relative-to "$REPOSITORY_DIR" "$SCRIPT_FILENAME")
if grep --quiet --line-regexp --fixed-strings --no-messages "$relative_script_filename" "$EXPECTED_FAILURES_LIST_FILENAME"; then
is_expected_to_fail=1
else
is_expected_to_fail=0
fi

# check the returncode of the test
if "$@" "$SCRIPT_FILENAME"; then
has_failed=0
else
has_failed=1
fi


# complain, if the result did not meet our expectation
if [ "$is_expected_to_fail" = "$has_failed" ]; then
# the check returned the expected result
exit 0
elif [ "$has_failed" = "1" ]; then
echo >&2 "ERROR: the script '$SCRIPT_FILENAME' should pass the test, but it failed"
exit 4
else
echo >&2 "ERROR: the script '$SCRIPT_FILENAME' was expected to fail the test, but it passed"
exit 5
fi

0 comments on commit e5abdee

Please sign in to comment.
You can’t perform that action at this time.