Analysis: Categorize LTE integ test failures

[LKreutzer]

Categorize LTE integ test failures

• Should cover the Make and Bazel workflows
• What are the most common failures?
• Discuss next steps

[LKreutzer]

• LTE integ test Bazel
  • Test failure
    • ⚠️ test_attach_detach_flaky_retry_success and following (ConnectionRefusedError: [Errno 111] Connection refused)
      • https://github.com/magma/magma/runs/8000898825?check_suite_focus=true
      • https://github.com/magma/magma/runs/7998802831?check_suite_focus=true
      • https://github.com/magma/magma/runs/7997793610?check_suite_focus=true
      • https://github.com/magma/magma/runs/7979184270?check_suite_focus=true
      • https://github.com/magma/magma/runs/7969551844?check_suite_focus=true
      • https://github.com/magma/magma/runs/7968170480?check_suite_focus=true
      • https://github.com/magma/magma/runs/7966899050?check_suite_focus=true
      • https://github.com/magma/magma/runs/7961392365?check_suite_focus=true
      • https://github.com/magma/magma/runs/7959464668?check_suite_focus=true
      • https://github.com/magma/magma/runs/7958058840?check_suite_focus=true
      • https://github.com/magma/magma/runs/7954780724?check_suite_focus=true
      • https://github.com/magma/magma/runs/7953613702?check_suite_focus=true
      • https://github.com/magma/magma/runs/7953613702?check_suite_focus=true
      • https://github.com/magma/magma/runs/7951362954?check_suite_focus=true
    • ⚠️ TestPagingAfterMmeRestart AssertionError: Timeout (60 sec) occurred while waiting for response message
      • https://github.com/magma/magma/runs/7993595303?check_suite_focus=true
      • https://github.com/magma/magma/runs/7993067621?check_suite_focus=true
      • https://github.com/magma/magma/runs/7991393262?check_suite_focus=true
      • https://github.com/magma/magma/runs/7985071514?check_suite_focus=true
      • https://github.com/magma/magma/runs/7952601098?check_suite_focus=true
      • https://github.com/magma/magma/runs/7952311249?check_suite_focus=true
      • https://github.com/magma/magma/runs/7952292053?check_suite_focus=true
      • https://github.com/magma/magma/runs/7950822132?check_suite_focus=true
  • Fatal error: local() encountered an error (return code 1) while executing 'vagrant up magma_test
    • https://github.com/magma/magma/runs/7991856851?check_suite_focus=true
    • https://github.com/magma/magma/runs/7950809879?check_suite_focus=true
    • This seems to be a bug in Vagrant
      • see description on: nil/NoMethodError in communicator.rb when executing several vagrant ups hashicorp/vagrant#12849
  • The hosted runner: GitHub Actions 17 lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
    • https://github.com/magma/magma/runs/7990209385?check_suite_focus=true
  • Sudo test failures
    • test_uplink_bridge
      • https://github.com/magma/magma/runs/7953770393?check_suite_focus=true
    • test_access_control
      • https://github.com/magma/magma/runs/7951134106?check_suite_focus=true
  • The requested URL returned error: 429 Fatal error: local() encountered an error (return code 1) while executing 'vagrant up magma'
    • Edit: This seems to be happening on the 1.8 branch, which still uses an older VM image that might not be in the cache anymore.
    • https://github.com/magma/magma/runs/8003520750?check_suite_focus=true
    • https://github.com/magma/magma/runs/8002041019?check_suite_focus=true
  • Ansible failed to complete successfully. The handshake operation timed out Install bazelisk Fatal error: local() encountered an error (return code 1) while executing 'vagrant up magma'
    • https://github.com/magma/magma/runs/8002950867?check_suite_focus=true
  • (:warning:) Bazel build fails ClosedChannelException DownloadTimeoutException 401 Unauthorized Authorization required
    • Edit: Assumption too many parallel runs. Might be a bottle neck in the remote cache access?
      • These runs where on the older VM, this might be the cause?
    • https://github.com/magma/magma/runs/8002038218?check_suite_focus=true (Wed, 24 Aug 2022 18:49:24 GMT)
    • https://github.com/magma/magma/runs/8002037077?check_suite_focus=true (Wed, 24 Aug 2022 18:49:17 GMT)
    • https://github.com/magma/magma/runs/8002036249?check_suite_focus=true (Wed, 24 Aug 2022 18:49:16 GMT)
    • https://github.com/magma/magma/runs/8002034182?check_suite_focus=true (Wed, 24 Aug 2022 18:49:09 GMT)

[LKreutzer]

Summary for this weeks failures of the LTE integ test Bazel workflow:

• Most common causes are test failures:
  • test_attach_detach_flaky_retry_success fails frequently
    • This is the first extended integ test.
    • Its failure leads to follow up failures.
    • Newly created in fix(agw): Fixed cleanup for successful integ test #13332 two weeks ago.
  • test_paging_after_mme_restart fails frequently
    • This is an extended integ test.
    • Last changes in the test file 13 months ago.

[LKreutzer]

• Bazel integ test job should not run on 1.8 branch - should only run on master until switchover.
  • PR chore(ci): Reconfigure the LTE integ test bazel workflow trigger to only run on master branch #13730

[LKreutzer]

• LTE integ test Make (only the failures on the master branch - the filter from GitHub does not seem to be working in this case)
  • Test failure
  • ⚠️ test_attach_detach_flaky_retry_success
    • https://github.com/magma/magma/runs/7969552511?check_suite_focus=true
    • https://github.com/magma/magma/runs/7966899104?check_suite_focus=true
    • https://github.com/magma/magma/runs/7961392307?check_suite_focus=true
    • https://github.com/magma/magma/runs/7959464807?check_suite_focus=true
    • https://github.com/magma/magma/runs/7958059107?check_suite_focus=true
    • https://github.com/magma/magma/runs/7954780865?check_suite_focus=true
    • https://github.com/magma/magma/runs/7953613923?check_suite_focus=true
    • https://github.com/magma/magma/runs/7952600703?check_suite_focus=true
  • test_3485_timer_for_default_bearer_with_mme_restart
    • https://github.com/magma/magma/runs/7950822118?check_suite_focus=true
  • test_tau_periodic_active AssertionError
    • https://github.com/magma/magma/runs/8002950753?check_suite_focus=true
    • https://github.com/magma/magma/runs/7948199499?check_suite_focus=true
  • test_nas_non_delivery_for_auth
    • https://github.com/magma/magma/runs/7997793622?check_suite_focus=true
  • test_attach_ul_udp_data_with_sessiond_restart
    • https://github.com/magma/magma/runs/7993595324?check_suite_focus=true
  • test_attach_ul_udp_data_with_mme_restart
    • https://github.com/magma/magma/runs/7993067314?check_suite_focus=true
    • https://github.com/magma/magma/runs/7985071522?check_suite_focus=true
  • test_attach_detach_rar_tcp_he
    • https://github.com/magma/magma/runs/7991856853?check_suite_focus=true
  • NoMethodError during vagrant up magma_test
    • https://github.com/magma/magma/runs/8002038157?check_suite_focus=true
    • https://github.com/magma/magma/runs/7991393553?check_suite_focus=true
    • other error during same command https://github.com/magma/magma/runs/7952310998?check_suite_focus=true
  • Timed out while waiting for the machine to boot. This means that Vagrant was unable to communicate with the guest machine within the configured ("config.vm.boot_timeout" value) time period. for magma_dev
    • https://github.com/magma/magma/runs/7952292069?check_suite_focus=true
  • Sudo test failures
    • test_uplink_bridge
      • https://github.com/magma/magma/runs/8000351515?check_suite_focus=true
    • test_access_control
      • https://github.com/magma/magma/runs/7998802782?check_suite_focus=true
      • https://github.com/magma/magma/runs/7951133906?check_suite_focus=true
    • test_ipv6_solicitation_5g
      • https://github.com/magma/magma/runs/7993019060?check_suite_focus=true
    • test_dhcp_client
      • https://github.com/magma/magma/runs/7953770657?check_suite_focus=true
  • The hosted runner: GitHub Actions 12 lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error
    • https://github.com/magma/magma/actions/runs/2913470892

[LKreutzer]

Summary for this weeks failures of the LTE integ test (Make) workflow:

• Most common cause are test failures
  • In particular the test_attach_detach_flaky_retry_success fails frequently (see above)
  • This is the first extended integ test.
  • Its failure leads to follow up failures.
  • Newly created in fix(agw): Fixed cleanup for successful integ test #13332 two weeks ago.

[LKreutzer]

Closing this issue as the analysis is complete and follow up has been created.