Merged
Conversation
…formance and error handling
…rness(langtest.py)
…rompt-handling-for-different-datasets User prompt handling for multi-dataset testing
…e harness reports over the timestamp.
…derboard functionality
…pdate leaderboard functionality
Bug fix/performance tests
…-augmentation-allow-access-without-harness-testing
…-importing-of-edited-testcases-into-harness Refactor: Improved the `import_edited_testcases()` functionality in Harness.
…pt-techniques Implementation of prompt techniques
…ization and readability.
…-benchmark-report Fix: Summary class to update summary dataframe and handle file path
…allow-access-without-harness-testing Refactor: Improve Code Organization and Readability
…-benchmark-report Improved: `rank_by` argument add to `harness.get_leaderboard()`
website updates
updated: langtest version in pip
ArshaanNazir
approved these changes
May 15, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📢 Highlights
John Snow Labs is excited to announce the release of LangTest 2.2.0! This update introduces powerful new features and enhancements to elevate your language model testing experience and deliver even greater insights.
🏆 Model Ranking & Leaderboard: LangTest introduces a comprehensive model ranking system. Use harness.get_leaderboard() to rank models based on various test metrics and retain previous rankings for historical comparison.
🔍 Few-Shot Model Evaluation: Optimize and evaluate your models using few-shot prompt techniques. This feature enables you to assess model performance with minimal data, providing valuable insights into model capabilities with limited examples.
📊 Evaluating NER in LLMs: This release extends support for Named Entity Recognition (NER) tasks specifically for Large Language Models (LLMs). Evaluate and benchmark LLMs on their NER performance with ease.
🚀 Enhanced Data Augmentation: The new DataAugmenter module allows for streamlined and harness-free data augmentation, making it simpler to enhance your datasets and improve model robustness.
🎯 Multi-Dataset Prompts: LangTest now offers optimized prompt handling for multiple datasets, allowing users to add custom prompts for each dataset, enabling seamless integration and efficient testing.