chore: Improve integ test wrapper script #13628

vktng · 2022-08-16T16:53:05Z

Summary

The integration test wrapper script for Bazel was lacking some desired features, this PR aims to resolve 4 of them:

The option --retry-on-failure and --retry-attempts N was added. With these you can rerun failing tests N times and they will only be considered failed if all attempts fail.
Multiple test targets can be specified. You can provide your own list of targets that can contain tests from all categories and the order does not matter.
After every run there will be a Junit XML report generated.
With the --rerun-previously-failed option you can rerun all previously failed tests.

Test Plan

Use options as described in bazel/scripts/run_integ_tests.sh --help.

Some examples:

Quick, only precommit tests:

bazel/scripts/run_integ_tests.sh  //lte/gateway/python/integ_tests/s1aptests:test_attach_detach //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_static_ip //lte/gateway/python/integ_tests/s1aptests:test_gateway_metrics_attach_detach //lte/gateway/python/integ_tests/s1aptests:test_attach_without_ips_available

More flaky and longer nonsanity tests:

bazel/scripts/run_integ_tests.sh  //lte/gateway/python/integ_tests/s1aptests:test_outoforder_erab_setup_rsp_dedicated_bearer  //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_multi_ue_looped //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_nw_triggered_delete_secondary_pdn //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_ipv6

Retry:

bazel/scripts/run_integ_tests.sh --retry-on-failure //lte/gateway/python/integ_tests/s1aptests:test_outoforder_erab_setup_rsp_dedicated_bearer --retry-attempts 1 //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_multi_ue_looped //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_nw_triggered_delete_secondary_pdn //lte/gateway/python/integ_tests/s1aptests:test_attach_detach_ipv6

Rerun:

bazel/scripts/run_integ_tests.sh  --rerun-previously-failed

Additional Information

This change is backwards-breaking

github-actions · 2022-08-16T16:53:24Z

Thanks for opening a PR! 💯

A couple initial guidelines

All commits must be signed off. This is enforced by DCO check.
All PR titles must follow the semantic commits format. This is enforced by Semantic PR.

Howto

Reviews. The "Reviewers" listed for this PR are the Magma maintainers who will shepherd it.
Checks. All required CI checks must pass before merge.
Merge. Once approved and passing CI checks, use the ready2merge label to indicate the maintainers can merge your PR.

More info

Please take a moment to read through the Magma project's

Contributing Conventions for norms around contributed code

If this is your first Magma PR, also consider reading

Developer Onboarding for onboarding as a new Magma developer
Development Workflow for guidance on your first pull request
CI Checks for points of contact for failing or flaky CI checks
GitHub-to-Slack mappings for Magma maintainers for guidance on how to contact maintainers on Slack

bazel/scripts/run_integ_tests.sh

github-actions · 2022-08-16T17:01:31Z

feg-workflow

    2 files 203 suites 40s ⏱️
374 tests 374 ✔️ 0 💤 0 ❌
388 runs 388 ✔️ 0 💤 0 ❌

Results for commit fee0c71.

♻️ This comment has been updated with latest results.

github-actions · 2022-08-16T17:38:50Z

dp-workflow

13 tests 13 ✔️ 2m 9s ⏱️
  1 suites   0 💤
  1 files   0 ❌

Results for commit fee0c71.

♻️ This comment has been updated with latest results.

github-actions · 2022-08-16T18:08:29Z

agw-workflow

615 tests 611 ✔️ 5m 21s ⏱️
    2 suites     4 💤
    2 files     0 ❌

Results for commit c7c861a.

♻️ This comment has been updated with latest results.

nstng

As discussed offline:

$ ./bazel/scripts/run_integ_tests.sh --retry-on-failure results in ./bazel/scripts/run_integ_tests.sh: line 111: POSITIONAL_ARGS: unbound variable
report/merged_report/report_all_tests.xml should not be localted under magma root, e.g., use something related to RESULTS_DIR := /var/tmp/test_results in make integ tests
-> in the repo is not good
fix result xml - make it publishable

vktng · 2022-08-22T14:57:04Z

As discussed offline:

* `$ ./bazel/scripts/run_integ_tests.sh --retry-on-failure` results in `./bazel/scripts/run_integ_tests.sh: line 111: POSITIONAL_ARGS: unbound variable`

* `report/merged_report/report_all_tests.xml` should not be localted under magma root, e.g., use something related to `RESULTS_DIR := /var/tmp/test_results` in make integ tests
  -> in the repo is not good

* fix result xml - make it publishable

Fixed.

sebathomas · 2022-08-22T15:06:14Z

lte/gateway/python/scripts/runtime_report.py

@@ -46,12 +46,15 @@ def merge_all_report(working_dir, list_xml_report_paths, output_path):
        xml_file_path = working_dir + "/" + xml_file_path
        test_result = ET.parse(xml_file_path)
        test_suites_data = test_result.getroot()
-
+        integration_tests = test_suites_data.attrib == {}


I'm a bit confused by the these changes. So if we're running integration tests, there's an empty dict here and the real data is in the first element? Maybe it would be good to add a comment.

When the xml is created with pytest then the structure is slightly different from what this script was intended to work with. If the attributes are missing, then we have to copy them from the child node.

I see. 👍

By the way, there's an unused variable in this function called c that you could remove if you like.

sebathomas

AGW changes looking good.

sebathomas · 2022-08-22T15:16:56Z

lte/gateway/python/scripts/runtime_report.py

@@ -46,12 +46,15 @@ def merge_all_report(working_dir, list_xml_report_paths, output_path):
        xml_file_path = working_dir + "/" + xml_file_path
        test_result = ET.parse(xml_file_path)
        test_suites_data = test_result.getroot()
-
+        integration_tests = test_suites_data.attrib == {}


I see. 👍

By the way, there's an unused variable in this function called c that you could remove if you like.

vktng · 2022-08-23T10:11:37Z

lte/gateway/python/integ_tests/s1aptests/BUILD.bazel

@@ -1199,10 +1198,7 @@ pytest_test(
    srcs = ["test_attach_detach_flaky_retry_success.py"],
    imports = [LTE_ROOT],
    tags = TAG_EXTENDED_TEST,
-    deps = [
-        ":s1ap_wrapper",
-        requirement("flaky"),


Since "flaky" is a dependency of pytest from now on you do not need to specify it. (It will lead to a duplication error.)
See bazel/python_test.bzl

sebathomas · 2022-08-24T08:53:35Z

Needs to be rebased to get the fix for the DP CI issue from master.

Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

…t wrapper script Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

… the wrapper script Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

vktng added the bazel changes for the Bazelification effort label Aug 16, 2022

vktng self-assigned this Aug 16, 2022

vktng requested a review from a team August 16, 2022 16:53

vktng requested a review from a team as a code owner August 16, 2022 16:53

vktng requested a review from jheidbrink August 16, 2022 16:53

pull-request-size bot added the size/L Denotes a Pull Request that changes 100-499 lines. label Aug 16, 2022

vktng requested a review from ajahl August 16, 2022 16:53

github-actions bot added the component: agw Access gateway-related issue label Aug 16, 2022

github-actions bot reviewed Aug 16, 2022

View reviewed changes

bazel/scripts/run_integ_tests.sh Outdated Show resolved Hide resolved

bazel/scripts/run_integ_tests.sh Outdated Show resolved Hide resolved

vktng force-pushed the improve_integ_test_wrapper_script branch from bb870b9 to 9dde152 Compare August 16, 2022 17:12

vktng linked an issue Aug 17, 2022 that may be closed by this pull request

Add improvements to the bazel/scripts/run_integ_tests.sh script #13525

Closed

vktng requested a review from ardzoht August 17, 2022 13:34

nstng requested changes Aug 22, 2022

View reviewed changes

vktng force-pushed the improve_integ_test_wrapper_script branch from 9dde152 to fe00aee Compare August 22, 2022 14:53

vktng requested a review from nstng August 22, 2022 14:54

LKreutzer approved these changes Aug 22, 2022

View reviewed changes

LKreutzer removed the request for review from jheidbrink August 22, 2022 14:55

vktng removed the request for review from ardzoht August 22, 2022 14:57

sebathomas reviewed Aug 22, 2022

View reviewed changes

sebathomas approved these changes Aug 22, 2022

View reviewed changes

vktng force-pushed the improve_integ_test_wrapper_script branch 3 times, most recently from 6a509f1 to c7c861a Compare August 23, 2022 10:07

vktng commented Aug 23, 2022

View reviewed changes

vktng added 4 commits August 24, 2022 10:55

chore: Add "retry on failure" option to integration test wrapper script

5f1f060

Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

chore: Add option to specify multiple targets for the integration tes…

8f614a1

…t wrapper script Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

chore: Add xml report generation to integration test wrapper script

5782268

Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

chore: Add option to rerun all previously failed integration tests to…

fee0c71

… the wrapper script Signed-off-by: Krisztián Varga <krisztian.varga@tngtech.com>

vktng force-pushed the improve_integ_test_wrapper_script branch from c7c861a to fee0c71 Compare August 24, 2022 08:56

LKreutzer enabled auto-merge (squash) August 24, 2022 09:37

nstng approved these changes Aug 24, 2022

View reviewed changes

LKreutzer merged commit b45c364 into magma:master Aug 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Improve integ test wrapper script #13628

chore: Improve integ test wrapper script #13628

vktng commented Aug 16, 2022

github-actions bot commented Aug 16, 2022

github-actions bot commented Aug 16, 2022 •

edited

github-actions bot commented Aug 16, 2022 •

edited

github-actions bot commented Aug 16, 2022 •

edited

nstng left a comment

vktng commented Aug 22, 2022

sebathomas Aug 22, 2022

vktng Aug 22, 2022

sebathomas Aug 22, 2022

sebathomas left a comment

sebathomas Aug 22, 2022

vktng Aug 23, 2022

sebathomas commented Aug 24, 2022

chore: Improve integ test wrapper script #13628

chore: Improve integ test wrapper script #13628

Conversation

vktng commented Aug 16, 2022

Summary

Test Plan

Additional Information

github-actions bot commented Aug 16, 2022

Howto

More info

github-actions bot commented Aug 16, 2022 • edited

feg-workflow

github-actions bot commented Aug 16, 2022 • edited

dp-workflow

github-actions bot commented Aug 16, 2022 • edited

agw-workflow

nstng left a comment

Choose a reason for hiding this comment

vktng commented Aug 22, 2022

sebathomas Aug 22, 2022

Choose a reason for hiding this comment

vktng Aug 22, 2022

Choose a reason for hiding this comment

sebathomas Aug 22, 2022

Choose a reason for hiding this comment

sebathomas left a comment

Choose a reason for hiding this comment

sebathomas Aug 22, 2022

Choose a reason for hiding this comment

vktng Aug 23, 2022

Choose a reason for hiding this comment

sebathomas commented Aug 24, 2022

github-actions bot commented Aug 16, 2022 •

edited

github-actions bot commented Aug 16, 2022 •

edited

github-actions bot commented Aug 16, 2022 •

edited