Artifactory Switch - MagmaCore to LinuxFondation

[nstng]

This issue describes the switch from the magma core (MC) artifactory to the linux foundation (LF) artifactory.

Background

At 2022.12 the license for the MC artifactory ends. Starting then an artifactory hosted by the LF will be used.

Requirements for the LF artifactory

• The community will not get credentials for the LF artifactory, this is, manual manipulations (add, edit, delete) will not be possible. All actions must happen via CI by credentials provided as git hub (GH) secrets.
• In problematic situations the LF support can do manual changes to the artifactory.
• For technical reasons the repositories need to be renamed (e.g., debian-test -> magma-packages-test). This implies that no redirect from the MC URL to the LF URL will be possible.

Migration Plan

• all artifacts in the MC artifactory are synced to the LF artifactory
• while the MC artifactory is still usable, all existing MC workflows are doing a double push to both artifactories
  • build_all.yml, bazel.yml
• all currently manual workflows for uploading artifacts in MC are implemented for CI
  • 3rd-party.yml, ovs
• as soon as possible ...
  • the push to the MC artifactory is removed
  • in master all references to the MC artifactory are changed to the LF artifactory
  • in release branches the MC artifactory is changes to the LF artifactory

Issues

01 High Prio
06 Middle Prio
11 Low Prio

Initial Sync

• 01 [REGISTRY][Phase1] Move all production packages to LF REgistry #13897
  • this only mentions *-prod repos - *-test should also be affected

Double Push

• 01 bazel debian artifacts are pushed to the LF artifactory #14445
• 01 [REGISTRY][Phase1] Every CI upload job should upload to the https://artifactory.magmacore.org/ui/packages and the LF registry #13898
• 11 3rd party chore(ci): Use new artifactory URL for third-party build #14545 - no double push, just switch to new artifactory
• 01 .github/workflows/agw-build-publish-container.yml
  • First part: chore: Revert to old tagging scheme; Run on PRs #14534
  • Push to new artifactory.
• 01 .github/workflows/deploy-build-from-pr.yml -> remove instead: chore: Do not publish build results from PRs #14507
• 11 adapt bazel.yml and build_all.yml for v1.8 and maybe v1.7 Fix artifactory publishing for old releases #14624

Add Missing Workflows

• 06 feat(ci): Added initial build and upload workflow #13431
• 06 feat(ci): Build and Publish 3rd party dependencies #14365
• 11 ovs Build OVS for new Artifactory in the CI #14572
• 11 all release promote workflows Migrate to Linux Foundation artifactory in promotion workflows #14590
  • .github/workflows/docker-promote.yml
  • .github/workflows/helm-promote.yml
• 11 Create a CI job to push new go versions to magma-blob in new artifactory #14529
• 11 drop hello world workflow and get related packages deleted Remove the dummy package workflow #14616

Testing

• 01 Test Magma build and installation with the new Artifactory #14272

Switch

• 06 change all endpoints on master chore: all artifactory endpoints on master are switched to lf #14508
• 06 change all endpoints on v1.7 and v1.8 v1.7 and v1.8 are using the lf artifactory for installing magma #14581
• 11 follow-up: check in GPG key instead of downloading from server Use checked-in GPG key instead of downloading it from Artifactory #14570
• 06 Remove the double push again #14531
• 06 Prepare notification of users to change sources.list

Ad Hoc Problems

• 01 A GPG key for the LF artifactory is configured #14500
  • Workaround: push dummy packages everywhere
• 01Docker URL issue
• 01 LF artifactory clean-up - remove not needed repositories
• 01 Virtual docker repos are configured for LF artifactory #14497
• 01 Remove docker-ci registry #14505
• 01 use latest tag for ghz images fix: Set latest tag for agwc images #14594

Open

• 01 artifactory is used in pre-burn VM images chore: the lf artifactory is setup during vm full_provision-ing #14514
  • 11 follow-up: create new base images with new artifactory baked in Create new VM base images #14541
• 01 build_all is flaky - size notification, idna issue
• 06 remove unused folders from artifactory https://magmacore.slack.com/archives/C03TN1LM8CB/p1670320477210849

Unsorted Issues

• Support the Artifactory migration #14453
• Rollout CI to upload artifacts to JFrog Artifactory for all repository types #13450
• Migrate JFrog Artifactory from Magma to Linux Foundation #13439

External Resources

• https://wiki.magmacore.org/display/HOME/2022-11-18+Artifactory+Status+Meeting
• Main discussion happens in Slack #questions-to-linux-foundation-it