Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 5 legalbench tasks (the 5 existing in HELM) #827

Merged
merged 8 commits into from
May 15, 2024
Merged

Conversation

perlitz
Copy link
Collaborator

@perlitz perlitz commented May 15, 2024

The LegalBench project is an ongoing open science effort to collaboratively curate tasks for evaluating legal reasoning in English large language models (LLMs). The benchmark currently consists of 162 tasks gathered from 40 contributors.

https://github.com/HazyResearch/legalbench/tree/main

Here, although adding only the 5 tasks appearing in HELM, we are also laying the foundations for adding many more of them.

Signed-off-by: Yotam Perlitz <yotam.perlitz@ibm.com>
Signed-off-by: Yotam Perlitz <yotam.perlitz@ibm.com>
Signed-off-by: Yotam Perlitz <yotam.perlitz@ibm.com>
Signed-off-by: Yotam Perlitz <yotam.perlitz@ibm.com>
…k to the catalog

Signed-off-by: Yotam Perlitz <yotam.perlitz@ibm.com>
Signed-off-by: Yotam Perlitz <yotam.perlitz@ibm.com>
@elronbandel elronbandel enabled auto-merge (squash) May 15, 2024 14:27
Copy link

codecov bot commented May 15, 2024

Codecov Report

Attention: Patch coverage is 75.00000% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 92.11%. Comparing base (5783d76) to head (71b4b42).
Report is 52 commits behind head on main.

Files Patch % Lines
src/unitxt/string_operators.py 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #827      +/-   ##
==========================================
+ Coverage   91.48%   92.11%   +0.63%     
==========================================
  Files         100      104       +4     
  Lines       10169    10695     +526     
==========================================
+ Hits         9303     9852     +549     
+ Misses        866      843      -23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@elronbandel elronbandel merged commit e5e5c30 into main May 15, 2024
8 of 9 checks passed
@elronbandel elronbandel deleted the add-legalbench branch May 15, 2024 14:49
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants