Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restructure BBQ data #725

Merged
merged 3 commits into from
Aug 31, 2023
Merged

Restructure BBQ data #725

merged 3 commits into from
Aug 31, 2023

Conversation

RakshitKhajuria
Copy link
Contributor

@RakshitKhajuria RakshitKhajuria commented Aug 25, 2023 •
edited

Description

Previously when evaluating the BBQ data there were some false negative results.

This PR aims at restructuring the BBQ data and refining the prompt to give better and accurate evaluations.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • This change requires a documentation update

Usage

harness = Harness(task="question-answering", model={"model": "text-davinci-003","hub":"openai"}, data={"data_source" :"BBQ-test-tiny"})

Screenshots (if appropriate):

@RakshitKhajuria RakshitKhajuria added 💡Enhancements Something can be improved v2.1.0 Issue or request to be done in v2.1.0 release labels Aug 25, 2023
@RakshitKhajuria RakshitKhajuria linked an issue Aug 25, 2023 that may be closed by this pull request
Improve BBQ dataset evaluation #648
Closed
@ArshaanNazir ArshaanNazir self-requested a review August 31, 2023 05:28
@ArshaanNazir ArshaanNazir merged commit e13717e into release/1.4.0 Aug 31, 2023
3 checks passed
@ArshaanNazir ArshaanNazir deleted the improve-bbq-data branch August 31, 2023 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ArshaanNazir ArshaanNazir ArshaanNazir approved these changes

@chakravarthik27 chakravarthik27 Awaiting requested review from chakravarthik27

Assignees

@RakshitKhajuria RakshitKhajuria

@Prikshit7766 Prikshit7766

Labels
💡Enhancements Something can be improved v2.1.0 Issue or request to be done in v2.1.0 release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improve BBQ dataset evaluation
3 participants
@RakshitKhajuria @ArshaanNazir @Prikshit7766