feat(agw): Create magma VM based on debian package #13552

voisey · 2022-08-09T15:52:00Z

Summary

Closes #13263.

Known issues associated with this PR:

In some cases, we see that pipelined is not reachable. This seems to be due to a problem with restarting this service in this set-up. This could be related to Fix mme memory leaks in debian production builds #13551.
agw_install_ubuntu.sh is basically duplicated into agw_install_ubuntu_vm.sh and agw_network_ubuntu.sh. The reason is that the agw_install_ubuntu.sh is the official installation script and it seems wrong to mix this with planned CI set-up. A flag was introduced into the magma_deploy role to get around this.

Test Plan

lte/gateway $ vagrant up magma_deb + same for magma_test + magma_trfserver + execute all relevant integration tests
All precommit and extended LTE integ tests ran green locally at least once

github-actions · 2022-08-09T15:52:19Z

Thanks for opening a PR! 💯

A couple initial guidelines

All commits must be signed off. This is enforced by DCO check.
All PR titles must follow the semantic commits format. This is enforced by Semantic PR.

Howto

Reviews. The "Reviewers" listed for this PR are the Magma maintainers who will shepherd it.
Checks. All required CI checks must pass before merge.
Merge. Once approved and passing CI checks, use the ready2merge label to indicate the maintainers can merge your PR.

More info

Please take a moment to read through the Magma project's

Contributing Conventions for norms around contributed code

If this is your first Magma PR, also consider reading

Developer Onboarding for onboarding as a new Magma developer
Development Workflow for guidance on your first pull request
CI Checks for points of contact for failing or flaky CI checks
GitHub-to-Slack mappings for Magma maintainers for guidance on how to contact maintainers on Slack

lte/gateway/deploy/roles/magma_deploy/vars/all_master.yaml

github-actions · 2022-08-09T16:02:37Z

feg-workflow

    2 files 203 suites 40s ⏱️
374 tests 374 ✔️ 0 💤 0 ❌
388 runs 388 ✔️ 0 💤 0 ❌

Results for commit 893a16d.

♻️ This comment has been updated with latest results.

github-actions · 2022-08-09T16:19:13Z

dp-workflow

13 tests 13 ✔️ 2m 52s ⏱️
  1 suites   0 💤
  1 files   0 ❌

Results for commit 893a16d.

♻️ This comment has been updated with latest results.

ssanadhya · 2022-08-09T16:41:58Z

lte/gateway/deploy/roles/magma_deb/files/post_install_changes.sh

+# TODO: GH13551. Ugly workaround to prevent service failures because of memory leaks 
+sed -i 's/\[Service\]/[Service]\nEnvironment="LSAN_OPTIONS=detect_leaks=0"/' /etc/systemd/system/magma@mme.service


The workaround suppresses all memory leaks, which is not a good idea. It will be better to suppress only the two known issues in libfluid and libczmq, but highlight if new memory leaks come up.

Thanks for the comment. In the end, we've decided to remove this line entirely since we've found that connection is lost with pipelined even with it. We propose to merge this PR as a first attempt and then to have a more detailed look at this in future PRs (already planned), when we try to use this in the CI to execute integration tests. Does this sound okay to you?

@voisey , do you plan to remove this line as part of this PR, or future PR?

I've now removed it from this PR.

github-actions · 2022-08-09T16:53:22Z

agw-workflow

615 tests 611 ✔️ 4m 41s ⏱️
    2 suites     4 💤
    2 files     0 ❌

Results for commit 893a16d.

♻️ This comment has been updated with latest results.

lte/gateway/deploy/roles/magma_deb/files/post_install_changes.sh

voisey · 2022-08-12T08:39:02Z

@nstng @ssanadhya The migration of the shell script to ansible is done, so this should be ready for review now.

nstng

lgtm - I can run integration tests vs magma_deb. I can reproduce that (very often) when one test fails then a couple follow up tests will fail because pipelined is not reachable. I agree that this issue can and shout be better analyzed when a setup is created to use magma_deb in CI.

ssanadhya · 2022-08-16T16:42:44Z

lte/gateway/deploy/roles/magma_deb/tasks/main.yml

+    - { src: 'orc8r/tools/ansible/roles/fluent_bit/files/magma_td-agent-bit.service', dest: 'magma@td-agent-bit.service' }
+    - { src: 'lte/gateway/deploy/roles/magma/files/systemd/magma_dp_envoy.service', dest: 'magma_dp@envoy.service' }
+
+- name: Reload magmad to apply service file changes


This step is redundant if you are eventually rebooting the system.

Thanks for spotting that, I've just removed it.

ssanadhya · 2022-08-16T16:45:04Z

@voisey , what's the purpose of 'magma_deploy' role?

ssanadhya · 2022-08-16T16:47:20Z

@voisey @nstng , FYI, this PR is also a step towards fixing #12075 . Can you add #12075 to your backlog too?

nstng · 2022-08-16T18:31:08Z

@voisey , what's the purpose of 'magma_deploy' role?

It's for installing a magma debian package (and some additional dependencies and configuration). The role is used in the official magma install script lte/gateway/deploy/agw_install_ubuntu.sh where the latest production magma.deb is installed (i.e. currently 1.7). Here the role is used to install the latest magma.deb created by the build_all workflow (i.e., the latest version in the magma artifactory in focal-ci).

ssanadhya

Lgtm, with minor comments

lte/gateway/Vagrantfile

Signed-off-by: Cameron Voisey <cameron.voisey@tngtech.com>

voisey requested a review from a team as a code owner August 9, 2022 15:52

voisey requested a review from uri200 August 9, 2022 15:52

pull-request-size bot added the size/L Denotes a Pull Request that changes 100-499 lines. label Aug 9, 2022

voisey mentioned this pull request Aug 9, 2022

chore(agw): Magma prod vm #13518

Closed

3 tasks

github-actions bot added component: agw Access gateway-related issue component: ci All updates on CI (Jenkins/CircleCi/Github Action) labels Aug 9, 2022

github-actions bot reviewed Aug 9, 2022

View reviewed changes

lte/gateway/deploy/roles/magma_deploy/vars/all_master.yaml Show resolved Hide resolved

voisey mentioned this pull request Aug 9, 2022

Fix mme memory leaks in debian production builds #13551

Closed

ssanadhya reviewed Aug 9, 2022

View reviewed changes

nstng self-requested a review August 9, 2022 17:48

voisey force-pushed the create_magma_prod_vm branch 2 times, most recently from 03611f1 to 6112bf0 Compare August 11, 2022 14:39

nstng requested changes Aug 11, 2022

View reviewed changes

lte/gateway/deploy/roles/magma_deb/files/post_install_changes.sh Outdated Show resolved Hide resolved

voisey force-pushed the create_magma_prod_vm branch 2 times, most recently from 6f38419 to 7e2cead Compare August 12, 2022 08:33

nstng self-requested a review August 15, 2022 12:37

nstng approved these changes Aug 16, 2022

View reviewed changes

voisey requested review from ssanadhya and removed request for uri200 August 16, 2022 07:32

ssanadhya reviewed Aug 16, 2022

View reviewed changes

voisey force-pushed the create_magma_prod_vm branch from 7e2cead to 9c005e9 Compare August 17, 2022 06:56

voisey requested a review from ssanadhya August 18, 2022 07:55

voisey mentioned this pull request Aug 18, 2022

Test v1.8 debian package #13650

Closed

4 tasks

ssanadhya approved these changes Aug 18, 2022

View reviewed changes

lte/gateway/Vagrantfile Show resolved Hide resolved

feat(agw): Create production-like magma VM for running integration tests

893a16d

Signed-off-by: Cameron Voisey <cameron.voisey@tngtech.com>

voisey force-pushed the create_magma_prod_vm branch from 9c005e9 to 893a16d Compare August 22, 2022 10:10

sebathomas approved these changes Aug 22, 2022

View reviewed changes

voisey merged commit 338e259 into magma:master Aug 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(agw): Create magma VM based on debian package #13552

feat(agw): Create magma VM based on debian package #13552

voisey commented Aug 9, 2022 •

edited by mpfirrmann

github-actions bot commented Aug 9, 2022

github-actions bot commented Aug 9, 2022 •

edited

github-actions bot commented Aug 9, 2022 •

edited

ssanadhya Aug 9, 2022

voisey Aug 10, 2022 •

edited

ssanadhya Aug 10, 2022

voisey Aug 11, 2022

github-actions bot commented Aug 9, 2022 •

edited

voisey commented Aug 12, 2022

nstng left a comment

ssanadhya Aug 16, 2022

voisey Aug 17, 2022

ssanadhya commented Aug 16, 2022

ssanadhya commented Aug 16, 2022

nstng commented Aug 16, 2022

ssanadhya left a comment

		# TODO: GH13551. Ugly workaround to prevent service failures because of memory leaks
		sed -i 's/\[Service\]/[Service]\nEnvironment="LSAN_OPTIONS=detect_leaks=0"/' /etc/systemd/system/magma@mme.service

feat(agw): Create magma VM based on debian package #13552

feat(agw): Create magma VM based on debian package #13552

Conversation

voisey commented Aug 9, 2022 • edited by mpfirrmann

Summary

Test Plan

github-actions bot commented Aug 9, 2022

Howto

More info

github-actions bot commented Aug 9, 2022 • edited

feg-workflow

github-actions bot commented Aug 9, 2022 • edited

dp-workflow

ssanadhya Aug 9, 2022

Choose a reason for hiding this comment

voisey Aug 10, 2022 • edited

Choose a reason for hiding this comment

ssanadhya Aug 10, 2022

Choose a reason for hiding this comment

voisey Aug 11, 2022

Choose a reason for hiding this comment

github-actions bot commented Aug 9, 2022 • edited

agw-workflow

voisey commented Aug 12, 2022

nstng left a comment

Choose a reason for hiding this comment

ssanadhya Aug 16, 2022

Choose a reason for hiding this comment

voisey Aug 17, 2022

Choose a reason for hiding this comment

ssanadhya commented Aug 16, 2022

ssanadhya commented Aug 16, 2022

nstng commented Aug 16, 2022

ssanadhya left a comment

Choose a reason for hiding this comment

voisey commented Aug 9, 2022 •

edited by mpfirrmann

github-actions bot commented Aug 9, 2022 •

edited

github-actions bot commented Aug 9, 2022 •

edited

voisey Aug 10, 2022 •

edited

github-actions bot commented Aug 9, 2022 •

edited