CI issues are analyzed and fixed or mitigated

[nstng]

Currently unstable or broken workflows on master:

• CWF integ tests
  • pre analysis: potentially green, but very very unstable
  • CWF integ test flakyness is mitigated #12977
  • Currently in progress by (Christian Krämer, Sagar Arora)
• LTE integ tests
  • pre analysis: potentially green, but unstable
  • LTE integ test flakyness is mitigated #13004
  • Will be analyzed by Ankit Kumar Aman
• agw build
  • broken python package process
    • python dependencies can not be packaged with fpm - workaround/alternative applied fix(ci): fix dependency issues in agw build #12994
      • OPEN (but not blocking): this should be solved for new python package versions!
  • broken magma.deb
    • see Py protobuf files in magma debian build are created #13033
    • approach for fix: fix(ci): fix dependency issues in agw build #12994
    • approach worked for one run, but then agw build: apt python packages do not clash with pip packages #13075 (PR fix(agw): clash of installed pip packages and to be installed apt packages is resolved #13077)
  • in general: unclear if "stateful" artifactory setup is feasible - input artifactory equals output artifactory
    • before the fb artifactory was the input and the magma artifactory was the output
• agw-workflow
  • MME test job: Test 11 is very flaky, see mme: Potentially flaky unit test #11362 (comment) (open/up for grabs)
• 5g tests
  • broken: current issue is the broken magma.deb from the agw build
• General
  • mitigate 429 rate limiting failures of vagrant cloud for builds using VMs - See
    • chore(ci): cache Vagrant boxes with Github Actions Cache #13003,
    • fix(ci): Work around unattended upgrades holding lock #13015 and
    • fix(ci): base box caching is refactored #13007
  • Investigate feg integration tests - Ci add federated integ test #13012
  • reduce logging in order to speed up build: chore(ci): excessive logging of asn1 builds is mitigated #13022
  • OPEN (not blocking, but polluting): unattended upgrade failures

[nstng]

closed as outdated - see #13954 for a collection of current CI flaky behavior/failures