building notebooks in parallel for tests & publish

[Borda]

🚀 Feature

Dynamically extend the time limit for running CI on PR touching more content

Motivation

do not penalise PR which are touching multiple notebooks

Additional context

Run one tiny job which would determine how many notebooks are changed, and pass this count N to the test job and for example set timeout for testing as N*30min

[Borda]

could be useful for Azure: https://gaunacode.com/passing-variables-between-jobs-for-azure-devops-pipelines

[Borda]

another an even better way would be building notebooks in parallel
similar to Lightning-AI/torchmetrics#764 and Lightning-AI/ecosystem-ci#18

For publishing, it is rather simple, as the git merge goes in CPU job, and each consecutive notebook will be its own job with...
For testing it is rather tricky as the testing shall be required check so not sure yet how to include all dynamic jos... 🐰

[Borda]

yep, so the initial job will generate a list of needed builds and handle eventual removals, and then all particular jobs could even have larger customization such as OS or Python versions... 🦦

Multi-job configuration

• https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops#multi-job-configuration