Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor/add configurable parameters #866

Merged
merged 17 commits into from
Nov 3, 2023
Merged

Refactor/add configurable parameters #866

merged 17 commits into from
Nov 3, 2023

Conversation

alytarik
Copy link
Contributor

@alytarik alytarik commented Oct 30, 2023 •
edited

Description

This PR adds configurable parameters instead of some hardcoded values in

  • CrowS-Pairs
    • diff_treshold
    • filter_treshold
  • Wino-Bias
    • diff_treshold
  • Clinical, Disinformation, Factuality,Translation
    • sentence_transformer

➤ Fixes #857

Type of change

  • New feature (non-breaking change which adds functionality)

Usage

tests:
  wino-bias:
    gender-occupational-stereotype:
      min_pass_rate: 0.70
      diff_threshold: 0.03
tests:
  crows-pairs:
    common-stereotypes:
      min_pass_rate: 0.70
      diff_threshold: 0.10
      filter_threshold: 0.15

Checklist:

  • I've added Google style docstrings to my code.
  • I've used pydantic for typing when/where necessary.
  • I have linted my code
  • I have added tests to cover my changes.

Screenshots (if appropriate):

@alytarik alytarik self-assigned this Oct 30, 2023
@alytarik alytarik linked an issue Oct 30, 2023 that may be closed by this pull request
Make parameters configurable #857
Closed
@alytarik alytarik marked this pull request as ready for review October 30, 2023 08:10
@alytarik alytarik added 💡Enhancements Something can be improved v2.1.0 Issue or request to be done in v2.1.0 release labels Oct 30, 2023
chakravarthik27
chakravarthik27 reviewed Oct 30, 2023
View reviewed changes
langtest/transform/__init__.py Outdated
@@ -1736,9 +1736,14 @@ def transform(self) -> List[Sample]:
Empty list

"""
print(self.tests)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we need this print()

Copy link
Contributor Author

@alytarik alytarik Oct 30, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testing purposes, removed now

@alytarik alytarik changed the base branch from refactor/delete-dead-code to release/1.8.0 October 30, 2023 12:17
@alytarik alytarik changed the base branch from release/1.8.0 to refactor/delete-dead-code October 30, 2023 12:27
@alytarik alytarik changed the base branch from refactor/delete-dead-code to release/1.8.0 November 3, 2023 10:00
@alytarik alytarik merged commit 3ce94b8 into release/1.8.0 Nov 3, 2023
2 of 3 checks passed
@ArshaanNazir ArshaanNazir deleted the refactor/add-configurable-parameters branch November 16, 2023 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chakravarthik27 chakravarthik27 chakravarthik27 approved these changes

Assignees

@alytarik alytarik

Labels
💡Enhancements Something can be improved v2.1.0 Issue or request to be done in v2.1.0 release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make parameters configurable
2 participants
@alytarik @chakravarthik27