Skip to content

Add logic to stop a model evaluation run#3421

Merged
charisk merged 2 commits intomainfrom
charisk/stop-modeval-run
Mar 1, 2024
Merged

Add logic to stop a model evaluation run#3421
charisk merged 2 commits intomainfrom
charisk/stop-modeval-run

Conversation

@charisk
Copy link
Copy Markdown
Contributor

@charisk charisk commented Feb 29, 2024

Updated the ModelEvaluator logic to deal with stopping of an evaluation run. As part of this I extracted the logic that creates and cancels a variant analysis into a private runVariantAnalysis function.

The code isn't super straight forward so I've added a bunch of comments.

I've also started adding some tests for the ModelEvaluator but only focused on the stopping aspect for now.

I've done the following manual tests:

  • Stop evaluation as soon as it starts
  • Stop evaluation during query pack building (the query pack building continues and you can see output in the extension log but the notification goes away)
  • Stop evaluation during variant analysis
  • Stop variant analysis from the variant analysis results view page
  • Start/stop a few times in the same session

Checklist

  • CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
  • Issues have been created for any UI or other user-facing changes made by this pull request.
  • [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

@charisk charisk marked this pull request as ready for review February 29, 2024 11:36
@charisk charisk requested a review from a team as a code owner February 29, 2024 11:36
Copy link
Copy Markdown
Contributor

@shati-patel shati-patel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! In terms of sensible manual testing, things work generally as expected for me too.

(I haven't been able to test "Stop variant analysis from the variant analysis results view page" since we don't auto-open the view any more)


In terms of ✨ chaos ✨ testing: If I hammer the Evaluate/Stop Evaluation button enough, I can get into a place where everything lags, and then multiple evaluation runs happen and things get a bit confusing. It seems like we kick off multiple MRVA runs, but can only cancel the latest one. It's hard to see what's actually going on since we don't display the results anywhere, so I'm not too worried for now! 🙃

@charisk
Copy link
Copy Markdown
Contributor Author

charisk commented Mar 1, 2024

In terms of ✨ chaos ✨ testing: If I hammer the Evaluate/Stop Evaluation button enough, I can get into a place where everything lags, and then multiple evaluation runs happen and things get a bit confusing. It seems like we kick off multiple MRVA runs, but can only cancel the latest one. It's hard to see what's actually going on since we don't display the results anywhere, so I'm not too worried for now! 🙃

That's true, we do have a limitation that you can only cancel the last evaluation run (we create a new token so the old one gets disposed. That, in combination with the fact that query bundling isn't truly cancelable, make it laggy. Since we're working on a prototype phase I think this is okay for now. Thanks for testing! 😄

@charisk charisk merged commit 96b7722 into main Mar 1, 2024
@charisk charisk deleted the charisk/stop-modeval-run branch March 1, 2024 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants