Q: Run pipeline on non-model changes

[epa095]

As it is today, if one does a change which does not effect the model MSE then the "evaluate model" step cancels the job, and the whole pipeline fails. This feels a bit unfortunate when one is doing code-changes which does not change the model, e.g. modify the ACI configuration or change the serve script in some way. The two latter are examples where I would still want the "Deploy to ACI" step to continue. I know I can set ALLOW_RUN_CANCEL to false, but this has the effect of registering the model, which is not really what I want. What I would want is for the model to not be registered if its equal, but proceed with the deployment steps.

Any thoughts?

[j-so]

To deploy without training/registering a model, you can set the MODEL_BUILD_ID variable. The pipeline will only run deployment steps on whatever model was registered from the build ID given.

Doc for reference:
https://github.com/microsoft/MLOpsPython/blob/8fb12afe706c3408618e92fffcb65a9a894d9fd0/docs/getting_started.md#set-up-the-pipeline (very last line)

[epa095]

Thanks @j-so, but that is a manual step, which is not very "Ops-like" ;-) Many of our commits does not cause changes to the model, but we want automatic CI on them as well, I don't know whats the best way to do it. Here are some proposals:

• Only run the retraining if the commit has changes in paths which should affect the model (use azure devops paths directive).
• Calculate and save a checksum (sha) of the model pickle, and only make a new version of the model if the sha is changed. If it is unchanged then do not cancel the pipeline, but use the old (but identical) version of the model.

Other proposals are welcome :-)

[j-so]

I see - yes using that variable is manual.

We're actually working on creating a version of the CI/CD flow using two separate pipelines (one to train/eval/register and one to deploy to ACI/AKS/Webapp).

So one possibility is to use a path trigger on the deploy pipeline once that's available.