Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/crows pairs #808

Merged
merged 10 commits into from
Oct 2, 2023
Merged

Feature/crows pairs #808

merged 10 commits into from
Oct 2, 2023

Conversation

alytarik
Copy link
Contributor

@alytarik alytarik commented Oct 2, 2023

Description

This PR aims to include Crows-Pairs dataset in langtest. This test uses a masked sentence and two tokens to decide whether the model has bias for/against one choice.


Fixes #728

Type of change

  • New feature (non-breaking change which adds functionality)

Usage

Tutorial Notebook

harness = Harness(
    data={"data_source":"Crows-Pairs"},
    model={"model":"bert-base-uncased", "hub":"huggingface"},
    task="crows-pairs",
    config={
    "tests": {
        "defaults": {
            "min_pass_rate": 1.0
        },
        "crows-pairs": {
            "common-stereotypes": {
                "min_pass_rate": 0.70
            }
        }
    }
})

Checklist:

  • I've added Google style docstrings to my code.
  • I've used pydantic for typing when/where necessary.
  • I have linted my code
  • I have added tests to cover my changes.

Screenshots (if appropriate):

image image image

@ArshaanNazir ArshaanNazir linked an issue Oct 2, 2023 that may be closed by this pull request
@ArshaanNazir ArshaanNazir added ⭐ Feature Indicates new feature requests v2.1.0 Issue or request to be done in v2.1.0 release labels Oct 2, 2023
@ArshaanNazir ArshaanNazir self-requested a review October 2, 2023 18:40
@ArshaanNazir ArshaanNazir merged commit 87f9423 into release/1.6.0 Oct 2, 2023
3 checks passed
@ArshaanNazir ArshaanNazir deleted the feature/crows-pairs branch October 4, 2023 09:22
@chakravarthik27 chakravarthik27 removed the v2.1.0 Issue or request to be done in v2.1.0 release label Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
⭐ Feature Indicates new feature requests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Explore CrowS-Pairs
3 participants