[WIP] - Proof of concept - optimising CI build

[trallard]

Related to #1407

Note this is still very in WIP - but wanted to get a PR in to share with you all what I am thinking/working on

Notes:

• I am moving away from Make to using doit some of the advantages of this are:
  • Python-based so we get all the syntactic sugar
  • it allows for much more flexibility when it comes to set tasks, dependencies, and manipulating artefacts/build outputs
  • others projects in the Jupyter ecosystem have started adopting it for automation and to complement their CI workflows and it works pretty well

What I have done:

• started to adopt the workflow shown in Improve performance of CI system #1407
• added cache for the Docker build and for Python dependencies
• note that to reduce times and points of failure (plus getting a small POC I am currently only building + testing base-notebook)
• I have to do some debugging since when I run the GitHub actions workflow locally with act all the tests are working fine and passing, but when ran in GH actions they all seem to fail
  • Also the manifest creation is failing 🤷🏽 but will also need to debug ]

[mathbunnyru]

@trallard please ping me when this is ready to be reviewed or you need some help.