Normalize workflow naming and mage usage [WIP]

[Sharpz7]

This ticket will track potential things to cleanup in our CI.

A lot of this can be based on what is present in https://github.com/G-Research/fasttrackml/tree/main/.github/workflows

New Structure

The only yamls that should be in the main workflows folder are the ones with the on tag. The goal of this is to have a structure that is easier to follow.

I think we should have a https://github.com/armadaproject/armada/blob/master/.github/workflows/ci.yml for each "type" of CI job:

• ci.yml: handles the main CI stuff
• ci-release.yml: handles releases
• ci-page.yml: handles the website
• ci-clients.yml handles the clients 
• ci-third-party.yml: handles third-party elements (i.e. airflow)

We can then have other folders to keep everything else. I.e go, python etc. This should help CI look cleaner and keep to the DRY principle, and make it more "natural" to keep our CI consistent across everything. (Based on code here: m4rs-mt/ILGPU#1046)

Mage / Tox Changes

All tox commands should go through mage to make CI consistent. It also removed the problem of needing to be in a different folder for it to work. Everything should be runnable from the root of the repo.

Mage Update / Verify

These are targets that Kubernetes has. Verify is equivalent to running CI locally in full. Update is everything like docs, linting, builds, being done for you in one bundled command.

Lookout CI Additions

We should:

• Lint and build Lookout in CI
• Cache lookout builds
• Copy the build instead of re-building inside the image for the "prepare" job in CI.

[dave-gantenbein]

Won't do