Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/clinical tests #707

Merged
merged 29 commits into from
Aug 16, 2023
Merged

Feature/clinical tests #707

merged 29 commits into from
Aug 16, 2023

Conversation

ArshaanNazir
Copy link
Collaborator

@ArshaanNazir ArshaanNazir commented Aug 14, 2023

Description

This PR aims to add a new category of clinical tests. For now, we will be checking the demographic-bias for 3 medical files( datasets) that have been manually curated by our team. (Gastroenterology-files, Medical-files and Oromaxillofacial-files). The test assesses any demographic bias the model might exhibit when suggesting treatment plans for two patients with identical diagnoses


Fixes #668

Link to Notebook: Clinical-Tests

Usage

os.environ["OPENAI_API_KEY"] = <OPENAI_API_KEY>
harness = Harness(task="clinical-tests",model={"model": "text-davinci-003", "hub": "openai"},data = {"data_source": "Gastroenterology-files"})
h.generate().run().report()

Checklist:

  • I've added Google style docstrings to my code.
  • I've used pydantic for typing when/where necessary.
  • I have linted my code
  • I have added tests to cover my changes.

@ArshaanNazir ArshaanNazir linked an issue Aug 14, 2023 that may be closed by this pull request
@ArshaanNazir ArshaanNazir added the v2.1.0 Issue or request to be done in v2.1.0 release label Aug 15, 2023
@ArshaanNazir ArshaanNazir merged commit accc425 into release/1.3.0 Aug 16, 2023
@ArshaanNazir ArshaanNazir deleted the feature/clinical_tests branch August 22, 2023 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v2.1.0 Issue or request to be done in v2.1.0 release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support annotation team's dataset
2 participants