Improve Text2SQL Metrics: Refactoring, New Execution Metric, and Bug Fixes #1841
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
* Add Multi Turn Metrics Support Signed-off-by: elronbandel <elronbandel@gmail.com> * Add multi-turn metrics and templates support Signed-off-by: elronbandel <elronbandel@gmail.com> * Refactor MultiTurnMetric into GroupMetric for improved grouping and item identification Signed-off-by: elronbandel <elronbandel@gmail.com> * Remove duplicate import of dict_get and update line number in secrets baseline Signed-off-by: elronbandel <elronbandel@gmail.com> * Implement sequential success accuracy metric and refactor group reduction logic Signed-off-by: elronbandel <elronbandel@gmail.com> * Format Signed-off-by: elronbandel <elronbandel@gmail.com> * Some fixes Signed-off-by: elronbandel <elronbandel@gmail.com> * Rename Signed-off-by: elronbandel <elronbandel@gmail.com> --------- Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
* Increase token limit so the judge can get to the actual answer. Signed-off-by: Jonathan Bnayahu <bnayahu@il.ibm.com> * now with the json file Signed-off-by: Jonathan Bnayahu <bnayahu@il.ibm.com> --------- Signed-off-by: Jonathan Bnayahu <bnayahu@il.ibm.com>
…enabled. (#1834) * Added example of running inference with log probability Signed-off-by: Yoav Katz <katz@il.ibm.com> * Initial changes to support generated_text in meta data Signed-off-by: Yoav Katz <katz@il.ibm.com> * Added missing generated_text in llava models Signed-off-by: Yoav Katz <katz@il.ibm.com> * Fixed WMLInferenceEngineChat and improved tests Signed-off-by: Yoav Katz <katz@il.ibm.com> * Added print header to exmaple Signed-off-by: Yoav Katz <katz@il.ibm.com> * Added "text" and "logprob" to OpenAiInferenceEngine Signed-off-by: Yoav Katz <katz@il.ibm.com> * Reverted test question change * Updated tests Signed-off-by: Yoav Katz <katz@il.ibm.com> --------- Signed-off-by: Yoav Katz <katz@il.ibm.com> Co-authored-by: Elron Bandel <elronbandel@gmail.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com> Co-authored-by: Elron Bandel <elronbandel@gmail.com>
Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
elronbandel
requested changes
Jun 25, 2025
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
Signed-off-by: Oktie Hassanzadeh <hassanzadeh@us.ibm.com>
|
Thanks @elronbandel - I believe I've fixed the tests. I also added some more docs to the text2sql util functions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces improvements to the Text2SQL evaluation framework, focusing on flexibility, accuracy, and robustness of execution-based metrics.
Changes: