Pacific AI Releases LangTest 2.8.0: MedExQA and HeadQA Support, Unified text-generation Task for Hugging Face Models, Embedding Evaluation Fixes, and Security Enhancements

[chakravarthik27]

📢 Highlights

We're excited to announce the latest LangTest release, expanding clinical benchmark coverage, unifying Hugging Face model handling, and hardening the library and documentation website.

• 🩺 MedExQA: Medical Question Answering Benchmark:
  LangTest now supports MedExQA, a medical QA benchmark covering five underrepresented specialties: biomedical engineering, clinical laboratory science, clinical psychology, occupational therapy, and speech-language pathology.

• 🧠 HeadQA: Complex Healthcare Reasoning Benchmark:
  We've added support for HeadQA, a multiple-choice benchmark drawn from healthcare specialization exams. It covers medicine, nursing, pharmacology, psychology, biology, and chemistry.

• 🔄 Unified text-generation Task for Hugging Face Models:
  Translation, summarization, and text2text-generation tasks for Hugging Face models using the transformers package have been migrated to a single text-generation pathway. This removes per-task pipeline branching, aligns local model behavior with hosted LLM hubs, and makes it substantially simpler to add new Hugging Face models.

• 🧮 Embedding-Based Functionalities Restored on the Latest OpenAI SDK:
  Embedding-based functionalities have been migrated from the deprecated OpenAI interface to the current SDK client and its updated response format. Similarity scoring for embedding-driven metrics now works reliably on modern SDK versions, with no configuration changes needed on your side.

• 🔒 Security Enhancements:
  Vulnerabilities and security issues have been resolved across both the core library and the documentation website, reinforcing LangTest's overall stability and safety.

🔥 Key Enhancements

🩺 MedExQA: Medical Question Answering with Multiple Explanations

We've added support for MedExQA, a medical question-answering benchmark built around five underrepresented specialties: biomedical engineering, clinical laboratory science, clinical psychology, occupational therapy, and speech-language pathology.

Key Features:

• Covers five specialties that are typically missing from mainstream medical benchmarks.
• Works with the full robustness test suite to measure stability under input perturbations.
• Subset selection lets you target a single specialty or evaluate across all of them.

How it works:

#Import Harness from the LangTest library
from langtest import Harness
import os

os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

Prompt:

prompt = """
You are a medical expert specializing in clinical reasoning and medical knowledge.

You will be given:
1. A medical multiple-choice question.
2. Five answer options labeled A, B, C, D and E.

Your task is to:
- Select the single best answer based on established medical knowledge.
- Return only the corresponding option letter.

Rules:
- Output exactly one uppercase letter: A, B, C, D or E.
- Do not provide any explanation, reasoning, punctuation, or additional text.

Example:

Question:
What is the most common cause of hypothyroidism in the United States?

Options:
A) Iodine deficiency
B) Hashimoto's thyroiditis
C) Graves' disease
D) Thyroidectomy
E) Secondary hypothyroidism

Output(A, B, C, D or E):
B

Now answer the following question.

Question:
{question}

Options:
{options}

Output(A, B, C, D or E):

"""

Harness Setup:

harness = Harness(
    task="question-answering",
    model={
        "model": "gpt-5.6-luna", 
        "hub": "openai",
        "type": "chat"
    },
    data={"data_source": "MedExQA",
          "subset": "all",
          "split": "test"},
    config={
        "model_parameters": {
            "user_prompt": prompt
        },
        'tests': {
            'defaults': {
                'min_pass_rate': 0.65
            },
            'robustness': {
                'uppercase': {'min_pass_rate': 0.66},
                'lowercase': {'min_pass_rate': 0.66},
                'add_ocr_typo': {'min_pass_rate': 0.66},
                'dyslexia_word_swap': {'min_pass_rate': 0.60}
            }
        }
    }
)

Execution:

harness.generate().run().report()

🧠 HeadQA: Complex Healthcare Reasoning Benchmark

LangTest now supports HeadQA, a multiple-choice benchmark sourced from healthcare specialization exams, available in LangTest. The questions are drawn from professional certification exams across medicine, nursing, pharmacology, psychology, biology, and chemistry.

Key Features:

• Exam-grade questions spanning healthcare and life-science domains.
• Emphasizes multi-step clinical and scientific reasoning over memorization.
• English-language support for consistent evaluation across model hubs.
• Compatible with the robustness suite to measure stability under perturbed inputs.

How it works:

#Import Harness from the LangTest library
from langtest import Harness
import os

os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

Prompt:

prompt = """
You are a medical expert specializing in clinical reasoning and medical knowledge.

You will be given:
1. A medical multiple-choice question.
2. Five answer options labeled A, B, C, D, and E.

Your task is to:
- Select the single best answer based on established medical knowledge.
- Return only the corresponding option letter.

Rules:
- Output exactly one uppercase letter: A, B, C, D, or E.
- Do not provide any explanation, reasoning, punctuation, or additional text.

Example:

Question:
What is the most common cause of hypothyroidism in the United States?

Options:
A) Iodine deficiency
B) Hashimoto's thyroiditis
C) Graves' disease
D) Thyroidectomy
E) Secondary hypothyroidism

Output(A, B, C, D or E):
B

Now answer the following question.

Question:
{question}

Options:
{options}

Output(A, B, C, D or E):
"""

Harness Setup:

harness = Harness(
    task="question-answering",
    model={
        "model": "gpt-5.6-luna", 
        "hub": "openai",
        "type": "chat"
    },
    data={"data_source": "HeadQA",
          "split": "test"},
    config={
        "model_parameters": {
            "user_prompt": prompt
        },
        'tests': {
            'defaults': {
                'min_pass_rate': 0.65
            },
            'robustness': {
                'uppercase': {'min_pass_rate': 0.66},
                'lowercase': {'min_pass_rate': 0.66},
                'add_ocr_typo': {'min_pass_rate': 0.66},
                'dyslexia_word_swap': {'min_pass_rate': 0.60}
            }
        }
    }
)

Execution:

harness.generate().run().report()

🔄 Unified text-generation Task for Hugging Face Models

The translation, summarization, and text2text-generation tasks for Hugging Face transformers models have been consolidated into a single text-generation pipeline, giving one consistent interface across local and hosted models.

• Transformers v5 removed the corresponding pipeline() helpers; LangTest now handles seq2seq loading internally through AutoTokenizer and AutoModelForSeq2SeqLM.
• Fully internal refactor — no configuration changes needed, with local models now aligned to hosted LLM hub behavior.

🧮 OpenAI SDK v1 Migration for Embedding-Based Evaluation

Embedding-based functionalities have been migrated from the deprecated OpenAI interface to the current SDK client, restoring reliable similarity scoring for embedding-driven metrics.

• What broke: OpenaiEmbeddings relied on the pre-1.x openai.Embedding.create() endpoint and module-level openai.api_key assignment, both removed in the modern SDK — causing embedding functionalities to fail outright on current openai versions.
• What's fixed: The class now instantiates a proper SDK client with the API key and calls the current embeddings endpoint, with response parsing updated from dictionary subscripting to the SDK's typed response objects.
• No changes required on your side: Existing evaluation configs using embedding metrics work as before — single-string and batched list inputs both continue to return correctly shaped vectors.

🔒 Security Enhancements

Vulnerabilities and security issues have been resolved across both the core library and the documentation website, reinforcing LangTest's overall stability and safety.

• Core library: Flagged vulnerabilities addressed and affected dependencies updated to patched versions.
• Documentation website: Security issues on the docs site remediated, including dependency-level fixes.

---

What's Changed

• fix: embedding based evaluation using openai sdk by @chakravarthik27 in fix: embedding based evaluation using openai sdk #1229
• Fix/vulnerabilities and security issues by @chakravarthik27 in Fix/vulnerabilities and security issues #1233
• partial implementation from release/2.8.0 by @chakravarthik27 in partial implementation from release/2.8.0 #1237
• chore: updated dependencies and regenerated the poetry lock file by @chakravarthik27 in chore: updated dependencies and regenerated the poetry lock file #1238
• Release/2.8.0 by @chakravarthik27 in Release/2.8.0 #1239
• fix: update imports and dependencies to address vulnerabilities by @chakravarthik27 in fix: update imports and dependencies to address vulnerabilities #1241
• chore/update documentation and dev dependencies by @chakravarthik27 in chore/update documentation and dev dependencies  #1244
• Release/2.8.0 by @chakravarthik27 in Release/2.8.0 #1242
• feature/implement the medexqa by @chakravarthik27 in feature/implement the medexqa #1246
• feat: implement HeadQA by @chakravarthik27 in feat: implement HeadQA #1248
• Docs/headqa and medexqa tutorials by @chakravarthik27 in Docs/headqa and medexqa tutorials #1250
• feat: added the new datasets by @chakravarthik27 in feat: added the new datasets #1249
• Fix the vulnerabilities and security issues by @chakravarthik27 in Fix the vulnerabilities and security issues #1252
• fix/vulnerabilities-and-security-issues by @chakravarthik27 in fix/vulnerabilities-and-security-issues #1253

Full Changelog: 2.7.0...2.8.0

---

This discussion was created from the release Pacific AI Releases LangTest 2.8.0: MedExQA and HeadQA Support, Unified text-generation Task for Hugging Face Models, Embedding Evaluation Fixes, and Security Enhancements.