add a filter to wikitq #1547
Merged
add a filter to wikitq #1547
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tejaswini
pushed a commit
that referenced
this pull request
Jan 24, 2025
elronbandel
added a commit
that referenced
this pull request
Jan 26, 2025
* Renamed criterias in LLM-as-a-Judge metrics to criteria. * Reintroduced imports that were removed from llm_as_judge.py * Updated the examples documentation for LLM-as-a-judge * Added missing import * Fixed formatting using ruff * add a filter to wikitq (#1547) * Add text2sql tasks to unitxt (#1414) * add text2sql templates Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add data managment utility for text2sql Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add basic template Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add sql execution accuracy metric Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add text2sql execution accuracy metric Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add text2sql task Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * condition download in presence of a cache dir Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add init fille Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add processors Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add processors Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add basic template Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * change id to int Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * change notations in templates Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * push to catalog Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add evidence, remove SL Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * remove unued function, fix Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix imports from unitxt.text2sql Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * push to catalog Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix cache location Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add example Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix imports Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add func_timeout to test reqs Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix typing Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * change template name Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * push to catalog Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add req Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add local model option Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix databases download Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix databases download Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add loader limit ot make example faster Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix cache paths, avoid re-download Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add type schema Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * remove inports from inits Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add text2sql to inits Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * update card to use serializers Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add schema serializer Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add text2sql serializer to default template Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add schema to task Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * adjust templates to using serializer Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * adjust templates to using serializer Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix processor Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * remove target prefix from template Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add shuffle to bird Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add shuffle to bird Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * edit template Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * remove comment from init Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * clear processors code Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add option with ticks Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add anls metric Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add template Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * drop comment Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * remove recursion limit Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add loader_limit to example Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix recursion error Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * move import to withing metric Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * remove catalog files wo prepare Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix typing Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * change template im example Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * moving text2sql implementaion to the main src dir Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix imports Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix imports Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix imports Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix imports Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * import data_utils Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * fix formatting Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * refactor names Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add processors tests Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add more tests Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add tests Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * refactor: allow more data sources Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * allow db source input Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * organize imports Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * update example Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add db_type to task Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * format Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add db_type to task Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add local db definition ability Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add EE tests Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * add tests Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * rename file Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * rename file Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * update sql metric Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * rename file Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * refactor types, serializers and metric Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> --------- Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> * Add deduplicate operator (#1549) * Add deduplicate operator Signed-off-by: elronbandel <elronbandel@gmail.com> * Deduplicate MMLU Signed-off-by: elronbandel <elronbandel@gmail.com> * Update Deduplicate example in documentation for clarity Signed-off-by: elronbandel <elronbandel@gmail.com> * Deduplicate social iqa Signed-off-by: elronbandel <elronbandel@gmail.com> --------- Signed-off-by: elronbandel <elronbandel@gmail.com> * Fix the authentication problem (#1550) * Attach assitant answers to their origins with url link (#1528) Signed-off-by: elronbandel <elronbandel@gmail.com> * Add mtrag benchmark (#1548) * Add mtrag benchmark Signed-off-by: elronbandel <elronbandel@gmail.com> * Add multi_type_serializer for references and prediction fields in various JSON metrics Signed-off-by: elronbandel <elronbandel@gmail.com> * Remove unused TempOperator class and delete obsolete multi_turn.json task file Signed-off-by: elronbandel <elronbandel@gmail.com> --------- Signed-off-by: elronbandel <elronbandel@gmail.com> * Update end of year summary blog (#1552) Signed-off-by: elronbandel <elronbandel@gmail.com> * Update strategic focus section in 2024 summary blog to emphasize usability Signed-off-by: elronbandel <elronbandel@gmail.com> * Added missing import * Fix llm as judge example Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> * apply 'pre-commit run --all-files' Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> --------- Signed-off-by: Yotam-Perlitz <y.perlitz@ibm.com> Signed-off-by: elronbandel <elronbandel@gmail.com> Signed-off-by: Martín Santillán Cooper <msantillancooper@ibm.com> Co-authored-by: Tejaswini Pedapati <tejaswinip@us.ibm.com> Co-authored-by: ShirApp <58909189+ShirApp@users.noreply.github.com> Co-authored-by: Yotam Perlitz <perlitz@gmail.com> Co-authored-by: Elron Bandel <elronbandel@gmail.com> Co-authored-by: Elad <eladv@il.ibm.com> Co-authored-by: Martín Santillán Cooper <msantillancooper@ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
there are a few large tables, that cannot be added to the prompts (context window)