Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase probability of flake check to reveal flakes #9840

Merged
merged 2 commits into from
Jul 4, 2023
Merged

Increase probability of flake check to reveal flakes #9840

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6479aee
Update for ginkgo deprecated parameters
dhiller Jun 1, 2023
38111a2
Modify flake checker to increase probability of finding flakes
dhiller Jun 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
34 changes: 27 additions & 7 deletions automation/repeated_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,27 @@
#
set -euo pipefail

# According to https://dl.acm.org/doi/abs/10.1145/3476105
#
# * almost all flaky tests were independent of the execution platform, so environmental issues should be considered with higher priority
# * devs may be unaware of many flakes due to order-dependent tests
# * simple techniques of reversing or shuffling the test order may be more efficient and effective than more sophisticated approaches
# * “One study found that 88% of flaky tests were found to consecutively fail up to a maximum of five times before passing,
# though another reported finding new flaky tests even after 10,000 test suite runs.”
#
# Therefore we by default
# * use only latest kubevirtci provider,
# * run all changed tests five times and
# * randomize the test order each time

export TIMESTAMP=${TIMESTAMP:-1}

function usage {
cat <<EOF
usage: [NUM_TESTS=x] [NEW_TESTS=test1_test|...|testn_test] $0 [kubevirtci_provider[ kubevirtci_provider ...]]

run test lanes repeatedly using the set of test files that have been
changed or added since last merge commit, set NEW_TESTS to explicitly name the tests to run)
changed or added since last merge commit, set NEW_TESTS to explicitly name the tests to run

options:
NUM_TESTS how often each test lane is run, default is 3
Expand All @@ -36,11 +49,15 @@ usage: [NUM_TESTS=x] [NEW_TESTS=test1_test|...|testn_test] $0 [kubevirtci_provid
If /clonerefs is at work you need to provide a target commit, as then the latest commit is a
merge commit (resulting in no changes detected)

example:
examples:

1. NEW_TESTS='operator_test' ./automation/repeated_test.sh 'k8s-1.27'

runs tests/operator_test.go x times on kubevirtci provider k8s-1.27

NEW_TESTS='operator_test' ./automation/repeated_test.sh 'k8s-1.16.2'
2. NEW_TESTS='operator_test' ./automation/repeated_test.sh

runs tests/operator_test.go three times on kubevirtci provider k8s-1.16.2
runs tests/operator_test.go x times on latest kubevirtci provider found

EOF
}
Expand Down Expand Up @@ -101,7 +118,7 @@ if (( $# > 0 )); then
shift
done
else
mapfile -t TEST_LANES < <( find cluster-up/cluster -name 'k8s-[0-9].[0-9][0-9]' -print | sort -rV | grep -oE 'k8s.*' | head -3 )
mapfile -t TEST_LANES < <( find cluster-up/cluster -name 'k8s-[0-9].[0-9][0-9]' -print | sort -rV | grep -oE 'k8s.*' | head -1 )
fi
echo "Test lanes: ${TEST_LANES[*]}"
for lane in "${TEST_LANES[@]}"; do
Expand All @@ -128,7 +145,7 @@ if [[ -z "${NEW_TESTS}" ]]; then
fi
echo "Test files touched: $(echo ${NEW_TESTS} | tr '|' ',')"

NUM_TESTS=${NUM_TESTS-3}
NUM_TESTS=${NUM_TESTS-5}
echo "Number of per lane runs: $NUM_TESTS"

if should_skip_test_run_due_to_too_many_tests "${NEW_TESTS}"; then
Expand All @@ -152,7 +169,10 @@ for lane in "${TEST_LANES[@]}"; do
for i in $(seq 1 "$NUM_TESTS"); do
echo "test lane: $lane, run: $i"
make cluster-sync
ginko_params="-no-color -succinct -slow-spec-threshold=30s -focus=${NEW_TESTS} -skip=QUARANTINE -regexScansFilePath=true"
ginko_params="-no-color -succinct -skip=QUARANTINE -randomize-all"
xpivarc marked this conversation as resolved.
Show resolved Hide resolved
for test_file in $(echo ${NEW_TESTS} | tr '|' '\n'); do
ginko_params+=" -focus-file=${test_file}"
done
FUNC_TEST_ARGS="$ginko_params" make functest
if [[ $? -ne 0 ]]; then
echo "test lane: $lane, run: $i, tests failed!"
Expand Down
2 changes: 1 addition & 1 deletion hack/functests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ function functest() {
KUBEVIRT_FUNC_TEST_SUITE_ARGS="-skip-dual-stack-test ${KUBEVIRT_FUNC_TEST_SUITE_ARGS}"
fi

_out/tests/ginkgo -timeout=3h -r -slow-spec-threshold=60s "$@" _out/tests/tests.test -- -kubeconfig=${kubeconfig} -container-tag=${docker_tag} -container-tag-alt=${docker_tag_alt} -container-prefix=${functest_docker_prefix} -image-prefix-alt=${image_prefix_alt} -oc-path=${oc} -kubectl-path=${kubectl} -gocli-path=${gocli} -installed-namespace=${namespace} -previous-release-tag=${PREVIOUS_RELEASE_TAG} -previous-release-registry=${previous_release_registry} -deploy-testing-infra=${deploy_testing_infra} -config=${kubevirt_test_config} --artifacts=${ARTIFACTS} --operator-manifest-path=${OPERATOR_MANIFEST_PATH} --testing-manifest-path=${TESTING_MANIFEST_PATH} ${KUBEVIRT_FUNC_TEST_SUITE_ARGS} -virtctl-path=${virtctl_path} -example-guest-agent-path=${example_guest_agent_path}
_out/tests/ginkgo -timeout=3h -r "$@" _out/tests/tests.test -- -kubeconfig=${kubeconfig} -container-tag=${docker_tag} -container-tag-alt=${docker_tag_alt} -container-prefix=${functest_docker_prefix} -image-prefix-alt=${image_prefix_alt} -oc-path=${oc} -kubectl-path=${kubectl} -gocli-path=${gocli} -installed-namespace=${namespace} -previous-release-tag=${PREVIOUS_RELEASE_TAG} -previous-release-registry=${previous_release_registry} -deploy-testing-infra=${deploy_testing_infra} -config=${kubevirt_test_config} --artifacts=${ARTIFACTS} --operator-manifest-path=${OPERATOR_MANIFEST_PATH} --testing-manifest-path=${TESTING_MANIFEST_PATH} ${KUBEVIRT_FUNC_TEST_SUITE_ARGS} -virtctl-path=${virtctl_path} -example-guest-agent-path=${example_guest_agent_path}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify why the slow spec threshold is now 60 instead of 30?

Copy link
Contributor Author

@dhiller dhiller Jul 4, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag -slow-spec-threshold has been deprecated among others with v2.5, therefore I have removed it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad! I got confused with colors 🤦

}

if [ "$KUBEVIRT_E2E_PARALLEL" == "true" ]; then
Expand Down