Check 1.8.0 release branch for regression

[mpfirrmann]

Run tests with the 1.8 branch / 1.8 release artifacts to ensure the functionality with respect to 3rd party changes.

• Set up and tests containerized AGW with 1.8 release images by manually adding the LF artifactory as DOCKER_REGISTRY and 1.8 as IMAGE_VERSION in lte/gateway/docker/.env.
• Install and test 1.8 debian package on a bare-ubuntu machine (similar to the CI tests, can also be done locally with the setup script to mimic production).
• Run package job (Magma Build & Publish) build: Pin versions to make it possible to build all dependencies #15049
• Test helm charts
• Ensure that the backporting process is functional and all relevant workflows are enabled
  • Related to this is Check functionality of Bazel workflows on v1.8 #14937, but this may be out of scope and handled separately.

[LKreutzer]

Updated the title and content of this issue after discussion with @nstng.

[LKreutzer]

It might be worth back-porting the CI fixes for the save-state phase-out. I can see warnings on the v1.8 branch, e.g. here

Edit: And also the node.js 12 issue https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.

[LKreutzer]

FYI: I saw many failures of this job https://github.com/magma/magma/actions/runs/4074215488/jobs/7020519837, something might be broken for the Make AGW unit tests.

[MoritzThomasHuebner]

Update: I worked on this with @wolfseb who is currently on vacation.

• Install and test 1.8 debian package on a bare-ubuntu machine (similar to the CI tests, can also be done locally with the setup script to mimic production). -> We managed to get a machine running and checked the logs for any smoking gun errors. All error messages we found are also present on a magma_dev machine built from master.

• Run package job (Magma Build & Publish). -> We executed the steps locally and we obtain a magma*.deb. However, there are still some issues. When building the magma*.deb, some python dependencies are updated and we obtain an incomplete set of magma dependencies. We did not yet manage to install it on a bare magma_deb machine for testing.

  • Comment out magma repo in apt sources and do an apt update
  • Compare created packages with artifactory magma-packages-prod/focal-1.8.0

• Test helm charts. -> This point is still somewhat unclear. We started by going through the minikube deployment steps and successfully deployed the orc8r on the host machine. However, this is not yet the complete set of helm charts. We did not yet succeed in deploying the lte-orc8r helm charts. There are also helm charts in other components such as the feg which we have not yet deployed/tested.

• Ensure that the backporting process is functional and all relevant workflows are enabled -> We created a dummy backport PR that would trigger all relevant workflows.

[MoritzThomasHuebner]

FYI: I saw many failures of this job https://github.com/magma/magma/actions/runs/4074215488/jobs/7020519837, something might be broken for the Make AGW unit tests.

@LKreutzer on my backport branch it looks ok. Maybe the test is just flaky?
#14873 (comment)

[LKreutzer]

FYI: I saw many failures of this job https://github.com/magma/magma/actions/runs/4074215488/jobs/7020519837, something might be broken for the Make AGW unit tests.

@LKreutzer on my backport branch it looks ok. Maybe the test is just flaky? #14873 (comment)

Thank you for checking - yes it is quite possible that the Make tests are flaky. It is probably okay as it is - if you want to go in more depth I suppose we could run the tests locally repeatedly, but this may not be necessary.

[LKreutzer]

FYI: I saw many failures of this job https://github.com/magma/magma/actions/runs/4074215488/jobs/7020519837, something might be broken for the Make AGW unit tests.

@LKreutzer on my backport branch it looks ok. Maybe the test is just flaky? #14873 (comment)

Thank you for checking - yes it is quite possible that the Make tests are flaky. It is probably okay as it is - if you want to go in more depth I suppose we could run the tests locally repeatedly, but this may not be necessary.

Update:
I suspect that the sessiond_integ_test is flaky. This is where the Make workflow is failing. In the Bazel workflow this test fails 2/3 runs. But in Make there is no retry logic.

//lte/gateway/c/session_manager/test:sessiond_integ_test                  FLAKY, failed in 2 out of 3 in 60.0s

[MoritzThomasHuebner]

Some intermediary findings after running fab dev package locally without the LF artifactory. Some packages were only found locally and some only on the artifactory. Still needs more investigation:

Packages only found in LF artifactory:

• typing-extensions -> Version on packages.ubuntu.com
• openvswitch -> Version on packages.ubuntu.com
• systemd-python -> Version on packages.ubuntu.com
• attrs -> python-attrs on packages.ubuntu.com, but not python3. Also confusing because override section says build 19.3.0 and 20.3.0 is built.
• aioeventlet -> python-aioeventlet on packages.ubuntu.com, but not python3
• click -> Version on packages.ubuntu.com
• json-pointer -> Version on packages.ubuntu.com

Packages only found locally: (compared to magma-packages-prod/focal-1.8.0)

• aiosignal -> Only found in magma-packages-test and magma-packages-tmp for focal 1.8.0
• async-timeout -> Only found in magma-packages-test and magma-packages-tmp for focal 1.8.0
• frozenlist -> Only found in magma-packages-test and magma-packages-tmp for focal 1.8.0
• hpack -> Not on focal 1.8.0
• hyperframe -> Not on focal 1.8.0
• isort -> Not on focal 1.8.0
• js-regex -> Only found in magma-packages-test and magma-packages-tmp for focal 1.8.0
• lazy-object-proxy -> Not on focal 1.8.0
• msgpack -> Not on focal 1.8.0
• sortedcontainers -> Not on focal 1.8.0
• toml -> Not on focal 1.8.0
• wrapt -> Not on focal 1.8.0

[MoritzThomasHuebner]

Update:
I went through the following steps

• Start magma dev v1.8
• Disable lf artifactory && sudo apt-get update
• run pydep and build-magma.sh via modified fabfile
• Start magma_deb on master without installing magma
  • In lte/gateway/deploy/roles/magma_deploy/tasks/main.yml comment out 129-141 (magma install)
• Disable LF artifactory && sudo apt-get update
• Install created packages (sudo apt-get install ./*.deb)
• For each missing package
  • make sure the missing package exists in the ubuntu artifactory
  • if not: we need to have a closer look
  • elif yes: sudo apt download package
    -> Try again

Results:

• The python-*.deb packages seem to be self consistent. We can run sudo apt-get install ./python3-*.
• td-agent-bit and magma-sctpd can also be installed
• Trying to install magma_1.8.0*.deb fails due to the following missing dependencies:

         Depends: grpc-dev (>= 1.15.0) but it is not installable
         Depends: python3-aioeventlet but it is not installable
         Depends: libsystemd-dev but it is not going to be installed
         Depends: magma-cpp-redis but it is not installable
         Depends: libfolly-dev but it is not installable
         Depends: getenvoy-envoy but it is not installable
         Depends: sentry-native but it is not installable
         Depends: td-agent-bit (>= 1.7.8) but it is not installable
         Depends: bcc-tools but it is not installable
         Depends: oai-asn1c but it is not installable
         Depends: oai-gnutls (>= 3.1.23) but it is not installable
         Depends: oai-nettle (>= 1.0.1) but it is not installable
         Depends: prometheus-cpp-dev (>= 1.0.2) but it is not installable
         Depends: liblfds710 but it is not installable
         Depends: libczmq-dev (>= 4.0.2-7) but it is not going to be installed
         Depends: oai-freediameter (>= 0.0.2) but it is not installable
         Depends: magma-libfluid (>= 0.1.0.7) but it is not installable
         Depends: libopenvswitch (>= 2.15.4-9-magma) but it is not installable
         Depends: openvswitch-switch (>= 2.15.4-9-magma) but it is not going to be installed
         Depends: openvswitch-common (>= 2.15.4-9-magma) but it is not going to be installed
         Depends: openvswitch-datapath-dkms (>= 2.15.4-9-magma) but it is not installable

• After adding magma.list back to /etc/apt/sources.list.d/ and running sudo apt-get update, we are able to install magma_1.8.0*.deb.
• Running magma-start does start all the services
• All services appear to be running

[panyogesh]

Closing it as 1.8 is released