Stop the support for the Make build system and move over to Bazel

[jordanvrtanoski]

Problem

Currently Magma supports two build systems for the artefacts:

• Make
• Bazel

The transition to Bazel as a build system started almost a year ago and ad this moment, Bazel had overtaken the Make build system. The Make build system became a legacy system for the projects and at this moment we already have functionality which is only supported by the Bazel system. This creates discrepancy between the systems and creates a problem for the future development of the CI/CD. Magma community can no longer support both systems so at this moment, we are faced with the decision either to stop and abandon the move to Bazel system or discontinue the Make system.

Solution

The proposal is to deprecate and fully remove the make system from the master branch and starting from the upcoming release (1.9) to support only Bazel as the build system. This transition doesn't include the discussion of moving the build of the Docker containers to Bazel, rather only the use of Bazel as build system for the artefacts inside the containers which currently is the case with majority of the Docker containers.

The following steps are proposed:

• Mark Make as obsolete
• Update the CI/CD pipeline to send warning to PRs containing updates of Make artefacts
• Upon preparing the branching for the next release, start removing all the Make artefacts from the master branch
• Release 1.9 will not contain the Make artifacts

Non-goals

As mentioned, the building of Docker containers using the Bazel is not scope of this proposal.

[nstng]

Hi, thank you for the proposal, we completely support it. A couple of additions and a summary of the current state.

The Bazel effort for now only concerns the AGW, e.g., ORC8R, NMS, FEG and CWAG are not affected yet. This is also a non-goal for now.

For AGW Docker images, yes it is not planned to create Docker images with Bazel. This is possible (a PoC exists), but there is currently no developer or end user benefit that would warrant this effort. You already mentioned that artifacts that run in the Docker containers are built with Bazel, this is correct for the c/c++ services. For the Python and Go services no make or Bazel builds are involved.

About the current state:

• Everything that was possible with make is now also possible with Bazel.
• There are some make targets that do not need to be migrated. Make is not used for building in those cases, but only as a script wrapper, e.g., running linters or running the integration tests.
• A couple of testing CI workflows are already completely migrated and only use Bazel. This includes all AGW unit tests and coverage generation.
• The Debian AGW build is currently duplicated, i.e., make created artifacts are created (tagged as 1.8.0) and Bazel artifacts (tagged as 1.9.0).
• The respective integration tests on Debian artifacts are also duplicated.

What is missing:

• The FEG integration tests are still running on an environment that is based on an AGW build with make. A PoC to change this to an AGW setup with a Debian package is created, but waits for the ongoing fabric upgrade.
• A couple of smaller polishing issues are in progress - this includes usual build system maintenance topics. Those are topics that need to be taken over by the community in the future.

What is needed for a switch-over:

• General handover - present the current state in detail, document it (there is already a comprehensive documentation) and find a driver for the topic.
  • We propose to start with a Bazel deep-dive Show&Tell session.
• Removing duplicated make build targets and CI workflows. A draft pull request exists (but still gets extended) that can be used as a guide line.
• A couple of clean-up issues (tracked on GitHub) that are possible after the switch-over is complete. This includes removing technical debt and unneeded dependencies.

[lucasgonze]

What is the scope of the work for "What is missing" and "What is needed for a switch-over"? Is it in the budget? If not, how much additional budget is needed?

[nstng]

What is the scope of the work for "What is missing" and "What is needed for a switch-over"? Is it in the budget? If not, how much additional budget is needed?

Hi @lucasgonze,

• "What is missing"
  • The FEG part is because of the existing PoC straight forward and will be completed in scope of the community handover - in terms of the Bazel handover
  • The polishing parts will be in parts done in the same scope - leftovers will be documented and should be handled by the community (these also include maintenance unknows that are needed for either build system)
• "What is needed for a switch-over"
  • If there is a fast and positive decision for this proposal then parts can already be handled in the same scope as above - but here it makes sense if the community already participates in order to gain knowledge of the build system.

[jordanvrtanoski]

Covered under existing work:

• AGW

Components that aren't yet covered, but would be simple:

• Building the Docker continers

Not covered:

• Orchestrator
• CWG

[nstng]

Accepted in TSC meeting 2023 Feb 13th https://wiki.magmacore.org/display/HOME/February+13%2C+2023+Meeting