Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arg to force comparison and always register/deploy model #70

Merged
merged 2 commits into from
Aug 4, 2022

Conversation

jfomhover
Copy link
Contributor

@jfomhover jfomhover commented Aug 4, 2022

PR into Azure/mlops-v2

here's a summary of the issue:

  • model deployment in the training pipeline is conditioned to a test comparing the perf of the fine tuned model with the non-fine-tuned model
  • in theory the fine-tuned model should win, but we're using a small SKU to allow everyone to run this NLP scenario without specific quota, and we limited the number of data to 1000 and epochs to 5 so that the training pipeline would not take hours
  • when running under those conditions, it is likely both models will perform pretty much the same, making comparison between model uncertain
  • also the baseline (non-fine-tuned) model is already pretty good in itself, making this comparison uncertain as well

We're proposing to force the comparison by default to switch to an always-deploy scenario, but leave the pipeline as it is so that users can adapt to their needs more easily and see the full extent of the pipeline.

Checklist

I have:

  • read and followed the contributing guidelines

Changes

  • add argument in comparison script to force comparison and force register the model

@jfomhover jfomhover changed the base branch from main to main-jul31 August 4, 2022 15:26
@cindyweng cindyweng merged commit 564558e into main-jul31 Aug 4, 2022
@cindyweng cindyweng deleted the jfomhover/forcecomparisonarg branch September 13, 2022 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants