feat(agw): AGW container restart check their health status #14008

mpfirrmann · 2022-09-23T15:07:38Z

Summary

The LTE integration tests use sleep timer to wait for a magma services to become functional again after restarting them as part of several tests. These sleep timers are based on empirical data for the systemd-based AGW and are not necessarily well suited to account for the containerized AGW's docker container to become healty again after a restart.
Instead, we want to check the container health as an indicator to continue with the tests after a restart (and at the same time leave the existing workflow for the systemd-based AGW as it is for now).

Some special thoughts to be considered here:

The container corresponding to the mme service is called oai_mme.
The sctpd service does not have a magma@ prefix. Nevertheless, it is started via the magmad service.
We want to avoid a double waiting time resolving these special cases

The PR is separated into ten commits:

get_service_name_from_init_system() is added to handle the special naming cases.
disable_service uses the correct service name and works with docker and systemd.
enable_service uses the correct service name and works with docker and systemd.
is_service_active() uses the correct name and checks for systemctl is-active {service_name} / docker container health status is "healthy".
A function to check the health of ALL of magma's AGW docker containers is added (excl. magma's health container which is disabled for the tests).
The wait for restart function checks if all containers are healthy (docker) or waits the given wait_time for a systemd-based AGW.
All service restarts are handled by restart_services(["list", "of", "services"], time_wait: int).
After restarting all services we wait for all docker containers to become healthy.
We adapt the pipelined/datapath check for the containerized AGW.
The restarts of AGW containers follows THIS scheme.

Closes #12956.

Test Plan

In its current state, all pre-commit and extended S1AP integration tests pass locally using a containerized and a systemd-based AGW.
Tested with GitHub actions (ongoing) to show the tests are still working:
https://github.com/mpfirrmann/magma/actions/runs/3113678195

Additional Information

This change is backwards-breaking

github-actions · 2022-09-23T15:08:12Z

Thanks for opening a PR! 💯

A couple initial guidelines

All commits must be signed off. This is enforced by DCO check.
All PR titles must follow the semantic commits format. This is enforced by Semantic PR.

Howto

Reviews. The "Reviewers" listed for this PR are the Magma maintainers who will shepherd it.
Checks. All required CI checks must pass before merge.
Merge. Once approved and passing CI checks, use the ready2merge label to indicate the maintainers can merge your PR.

More info

Please take a moment to read through the Magma project's

Contributing Conventions for norms around contributed code

If this is your first Magma PR, also consider reading

Developer Onboarding for onboarding as a new Magma developer
Development Workflow for guidance on your first pull request
CI Checks for points of contact for failing or flaky CI checks
GitHub-to-Slack mappings for Magma maintainers for guidance on how to contact maintainers on Slack

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py

github-actions · 2022-09-23T15:17:08Z

feg-workflow

    2 files 203 suites 40s ⏱️
374 tests 374 ✔️ 0 💤 0 ❌
388 runs 388 ✔️ 0 💤 0 ❌

Results for commit 2a1653c.

♻️ This comment has been updated with latest results.

github-actions · 2022-09-23T15:34:00Z

dp-workflow

14 tests 14 ✔️ 2m 20s ⏱️
  1 suites   0 💤
  1 files   0 ❌

Results for commit 2a1653c.

♻️ This comment has been updated with latest results.

github-actions · 2022-09-23T15:57:02Z

agw-workflow

615 tests 611 ✔️ 3m 58s ⏱️
    2 suites     4 💤
    2 files     0 ❌

Results for commit 2a1653c.

♻️ This comment has been updated with latest results.

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py

VinashakAnkitAman

Left some comments

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py

MoritzThomasHuebner · 2022-10-05T05:29:26Z

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py

@@ -1002,17 +1002,18 @@ def restart_all_services(self):
                "sudo systemctl stop 'magma@*' 'sctpd' ;"
                "sudo systemctl start magma@magmad",
            )
+            self._wait_for_pipelined_to_initialize()
+
+            EXTRA_WAIT_TIME_FOR_OTHER_SERVICES_SECONDS = 10


Maybe we can collect the constants defined in this class and make the class attributes similar to how this is handled in S1ApUtil or SubscriberUtil

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

…ter `restart_all_services()` Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

VinashakAnkitAman

LGTM

mpfirrmann requested review from a team and rsarwad September 23, 2022 15:07

pull-request-size bot added the size/L Denotes a Pull Request that changes 100-499 lines. label Sep 23, 2022

github-actions bot added the component: agw Access gateway-related issue label Sep 23, 2022

github-actions bot reviewed Sep 23, 2022

View reviewed changes

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py Show resolved Hide resolved

mpfirrmann self-assigned this Sep 23, 2022

mpfirrmann linked an issue Sep 23, 2022 that may be closed by this pull request

Make AGW service restarts aware of container status #13965

Closed

4 tasks

mpfirrmann force-pushed the pr/docker_systemd_restart branch 4 times, most recently from af08b95 to 202384d Compare September 26, 2022 16:05

wolfseb reviewed Sep 27, 2022

View reviewed changes

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py Outdated Show resolved Hide resolved

mpfirrmann force-pushed the pr/docker_systemd_restart branch from 202384d to 86d2a47 Compare September 27, 2022 13:46

VinashakAnkitAman requested changes Sep 27, 2022

View reviewed changes

mpfirrmann force-pushed the pr/docker_systemd_restart branch 2 times, most recently from 41a610e to eba742f Compare September 28, 2022 09:44

nstng reviewed Sep 28, 2022

View reviewed changes

mpfirrmann requested a review from VinashakAnkitAman September 28, 2022 11:55

mpfirrmann force-pushed the pr/docker_systemd_restart branch from eba742f to e6b1187 Compare September 30, 2022 14:52

VinashakAnkitAman reviewed Oct 3, 2022

View reviewed changes

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py Show resolved Hide resolved

mpfirrmann force-pushed the pr/docker_systemd_restart branch from e6b1187 to 159431b Compare October 4, 2022 07:30

wolfseb requested changes Oct 5, 2022

View reviewed changes

lte/gateway/python/integ_tests/s1aptests/s1ap_utils.py Show resolved Hide resolved

wolfseb mentioned this pull request Oct 5, 2022

chore(agw): reduce python container build context #14085

Merged

1 task

MoritzThomasHuebner reviewed Oct 5, 2022

View reviewed changes

mpfirrmann force-pushed the pr/docker_systemd_restart branch from 159431b to 1883308 Compare October 5, 2022 08:30

feat(agw): get_service_name_from_init_system() is added

59f7720

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

mpfirrmann added 9 commits October 5, 2022 10:31

feat(agw): disable_service() works correctly with both init systems

327cbc7

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): enable_service() works correctly with both init systems

840a24e

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): is_service_active() works correctly with both init systems

baf382e

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): check if all AGW containers are healthy is added

ffc685e

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): wait for restart fct waits for healthy docker containers

d1bb77c

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): only wait once for restart

cf73912

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): dockerized AGW waits for all services to become healthy af…

b69ee40

…ter `restart_all_services()` Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): wait for pipelined is adapted for containerized AGW

55c7d00

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

feat(agw): docker restarts handle interdependencies manually

2a1653c

Signed-off-by: Marco Pfirrmann <marco.pfirrmann@tngtech.com>

mpfirrmann force-pushed the pr/docker_systemd_restart branch from 1883308 to 2a1653c Compare October 5, 2022 08:31

mpfirrmann requested review from wolfseb and VinashakAnkitAman October 5, 2022 12:31

wolfseb approved these changes Oct 6, 2022

View reviewed changes

magma deleted a comment from github-actions bot Oct 10, 2022

VinashakAnkitAman approved these changes Oct 10, 2022

View reviewed changes

mpfirrmann merged commit 638acb5 into magma:master Oct 11, 2022

mpfirrmann deleted the pr/docker_systemd_restart branch October 11, 2022 06:20

This was referenced Oct 12, 2022

chore(agw): multiple magma services can be enabled/disabled at the same time #14141

Merged

feat(agw): add service function to get all magma services #14142

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(agw): AGW container restart check their health status #14008

feat(agw): AGW container restart check their health status #14008

mpfirrmann commented Sep 23, 2022 •

edited

github-actions bot commented Sep 23, 2022

github-actions bot commented Sep 23, 2022 •

edited

github-actions bot commented Sep 23, 2022 •

edited

github-actions bot commented Sep 23, 2022 •

edited

VinashakAnkitAman left a comment

MoritzThomasHuebner Oct 5, 2022

VinashakAnkitAman left a comment

feat(agw): AGW container restart check their health status #14008

feat(agw): AGW container restart check their health status #14008

Conversation

mpfirrmann commented Sep 23, 2022 • edited

Summary

Test Plan

Additional Information

github-actions bot commented Sep 23, 2022

Howto

More info

github-actions bot commented Sep 23, 2022 • edited

feg-workflow

github-actions bot commented Sep 23, 2022 • edited

dp-workflow

github-actions bot commented Sep 23, 2022 • edited

agw-workflow

VinashakAnkitAman left a comment

Choose a reason for hiding this comment

MoritzThomasHuebner Oct 5, 2022

Choose a reason for hiding this comment

VinashakAnkitAman left a comment

Choose a reason for hiding this comment

mpfirrmann commented Sep 23, 2022 •

edited

github-actions bot commented Sep 23, 2022 •

edited

github-actions bot commented Sep 23, 2022 •

edited

github-actions bot commented Sep 23, 2022 •

edited