PacificAI · ArshaanNazir · Aug 3, 2023 · Jul 5, 2023 · Jul 5, 2023 · Jul 10, 2023
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -0,0 +1,32 @@
+name: Release
+on:
+  release:
+    types:
+      - published
+
+jobs:
+  publish:
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [3.9]
+        poetry-version: [1.3.1]
+        os: [ubuntu-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+      - uses: snok/install-poetry@v1
+        with:
+          version: ${{ matrix.poetry-version }}
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+      - name: Publish langtest ${{ github.event.release.name }} to pypi
+        env:
+          PYPI_SECRET: ${{ secrets.PYPI_SECRET }}
+        run: |
+          poetry config pypi-token.pypi $PYPI_SECRET
+          poetry publish --build
diff --git a/README.md b/README.md
@@ -40,7 +40,7 @@ Take a look at our official page for user documentation and examples: [langtest.
 
 ```python
 # Install langtest
-!pip install langtest transformers==4.28.1
+!pip install langtest[transformers]
 
 # Import and create a Harness object
 from langtest import Harness
@@ -67,6 +67,27 @@ We propose here an early stage open-source community project that aims to fill t
 
 [John Snow Labs](www.johnsnowlabs.com) has a full development team allocated to the project and is committed to improving the library for years, as we do with other open-source libraries. Expect frequent releases with new test types, tasks, languages, and platforms to be added regularly. We look forward to working together to make safe, reliable, and responsible NLP an everyday reality. 
 
+## Comparing Benchmark Datasets: Use Cases and Evaluations
+
+Langtest comes with different datasets to test your models, covering a wide range of use cases and evaluation scenarios.
+
+| Dataset       | Use Case                                                                                           | Notebook                                                                                                                                             |
+|---------------|----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [**BoolQ**](https://aclanthology.org/N19-1300/)    | Evaluate the ability of your model to answer boolean questions (yes/no) based on a given passage or context.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/BoolQ_dataset.ipynb)   |
+| [**NQ-open**](https://aclanthology.org/Q19-1026/)   | Evaluate the ability of your model to answer open-ended questions based on a given passage or context.  | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/NQ_open_dataset.ipynb)   |
+| [**TruthfulQA**](https://aclanthology.org/2022.acl-long.229/) | Evaluate the model's capability to answer questions accurately and truthfully based on the provided information.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb)   |
+| [**MMLU**](https://arxiv.org/abs/2009.03300)      | Evaluate language understanding models' performance in different domains. It covers 57 subjects across STEM, the humanities, the social sciences, and more.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/mmlu_dataset.ipynb)   |
+| [**NarrativeQA**](https://aclanthology.org/Q18-1023/) | Evaluate your model's ability to comprehend and answer questions about long and complex narratives, such as stories or articles.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb)   |
+| [**HellaSwag**](https://aclanthology.org/P19-1472/) | Evaluate your model's ability in completions of sentences.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb)   |
+| [**Quac**](https://aclanthology.org/D18-1241/)      | Evaluate your model's ability to answer questions given a conversational context, focusing on dialogue-based question-answering.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb)   |
+| [**OpenBookQA**](https://allenai.org/data/open-book-qa)| Evaluate your model's ability to answer questions that require complex reasoning and inference based on general knowledge, similar to an "open-book" exam.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/quac_dataset.ipynb)   |
+| [**BBQ**](https://arxiv.org/abs/2110.08193)       | Evaluate how your model responds to questions in the presence of social biases against protected classes across various social dimensions. Assess biases in model outputs with both under-informative and adequately informative contexts, aiming to promote fair and unbiased question-answering models.   | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/BBQ_dataset.ipynb)   |
+|[**XSum**](https://aclanthology.org/D18-1206/) | Evaluate your model's ability to generate concise and informative summaries for long articles with the XSum dataset. It consists of articles and corresponding one-sentence summaries, offering a valuable benchmark for text summarization models. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/XSum_dataset.ipynb)|
+|[**Real Toxicity Prompts**](https://aclanthology.org/2020.findings-emnlp.301/) | Evaluate your model's accuracy in recognizing and handling toxic language with the Real Toxicity Prompts dataset. It contains real-world prompts from online platforms, ensuring robustness in NLP models to maintain safe environments. | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/OpenAI_QA_Testing_Notebook.ipynb)
+
+> **Note**
+> For usage and documentation, head over to [langtest.org](https://langtest.org/docs/pages/docs/data#question-answering)
+
 ## Contributing
 
 We welcome all sorts of contributions:

diff --git a/demo/tutorials/end-to-end-notebooks/HuggingFace_Real_World_Notebook.ipynb b/demo/tutorials/end-to-end-notebooks/HuggingFace_Real_World_Notebook.ipynb
@@ -1412,7 +1412,11 @@
         }
       ],
       "source": [
-        "h.augment(input_path=\"conll03.conll\", output_path=\"augmented_conll03.conll\", inplace=False)"
+        "data_kwargs = {\n",
+        "      \"data_source\" : \"conll03.conll\",\n",
+        "       }\n",
+        "\n",
+        "h.augment(training_data=data_kwargs, save_data_path=\"augmented_conll03.conll\", export_mode=\"add\")"
       ]
     },
     {

diff --git a/demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Custom_Pipeline_Notebook.ipynb b/demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Custom_Pipeline_Notebook.ipynb
@@ -1368,7 +1368,11 @@
         }
       ],
       "source": [
-        "harness.augment(\"conll03.conll\", \"augmented_train.conll\", inplace=False)"
+        "data_kwargs = {\n",
+        "      \"data_source\" : \"conll03.conll\",\n",
+        "       }\n",
+        "\n",
+        "harness.augment(training_data=data_kwargs, save_data_path=\"augmented_train.conll\", export_mode=\"add\")"
       ]
     },
     {

diff --git a/demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Notebook.ipynb b/demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Notebook.ipynb
@@ -1203,7 +1203,11 @@
         }
       ],
       "source": [
-        "harness.augment(\"conll03.conll\", \"augmented.conll\", inplace=False)"
+        "data_kwargs = {\n",
+        "      \"data_source\" : \"conll03.conll\",\n",
+        "       }\n",
+        "\n",
+        "harness.augment(training_data=data_kwargs, save_data_path=\"augmented.conll\", export_mode=\"add\")"
       ]
     },
     {

diff --git a/demo/tutorials/end-to-end-notebooks/Spacy_Real_World_Notebook.ipynb b/demo/tutorials/end-to-end-notebooks/Spacy_Real_World_Notebook.ipynb
@@ -1382,7 +1382,11 @@
         }
       ],
       "source": [
-        "h.augment(\"conll03.conll\", \"augmented.conll\", inplace=False)"
+        "data_kwargs = {\n",
+        "      \"data_source\" : \"conll03.conll\",\n",
+        "       }\n",
+        "\n",
+        "h.augment(training_data=data_kwargs, save_data_path=\"augmented.conll\", export_mode=\"add\")"
       ]
     },
     {

diff --git a/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb
@@ -17,7 +17,7 @@
         "id": "3o5sAOfwL5qd"
       },
       "source": [
-        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/AI21_QA_Testing_Notebook.ipynb)"
+        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb)"
       ]
     },
     {
@@ -39,7 +39,7 @@
         "id": "26qXWhCYhHAt"
       },
       "source": [
-        "# Getting started with LangTest on John Snow Labs"
+        "# Getting started with LangTest"
       ]
     },
     {
@@ -54,23 +54,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install langtest"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Installing required dependencies"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "!pip install \"langtest[evaluate,ai21,langchain]\" transformers==4.28.1"
+        "!pip install \"langtest[evaluate,ai21,langchain,transformers]\" "
       ]
     },
     {

diff --git a/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb
@@ -17,7 +17,7 @@
         "id": "3o5sAOfwL5qd"
       },
       "source": [
-        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Testing_Notebook.ipynb)"
+        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb)"
       ]
     },
     {
@@ -39,29 +39,7 @@
         "id": "26qXWhCYhHAt"
       },
       "source": [
-        "# Getting started with LangTest on John Snow Labs"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "azUb114QhOsY",
-        "outputId": "45a0a692-5076-4fcf-ceb4-b5eb914d9aed"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install langtest"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Installing required dependencies"
+        "# Getting started with LangTest"
       ]
     },
     {
@@ -70,7 +48,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "!pip install \"langtest[evaluate,langchain,openai]\" transformers==4.28.1"
+        "!pip install \"langtest[evaluate,langchain,openai,transformers]\" "
       ]
     },
     {

diff --git a/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb
@@ -17,7 +17,7 @@
         "id": "3o5sAOfwL5qd"
       },
       "source": [
-        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Cohere_QA_Testing_Notebook.ipynb)"
+        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Cohere_QA_Summarization_Testing_Notebook.ipynb)"
       ]
     },
     {
@@ -39,29 +39,7 @@
         "id": "26qXWhCYhHAt"
       },
       "source": [
-        "# Getting started with LangTest on John Snow Labs"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "colab": {
-          "base_uri": "https://localhost:8080/"
-        },
-        "id": "azUb114QhOsY",
-        "outputId": "45a0a692-5076-4fcf-ceb4-b5eb914d9aed"
-      },
-      "outputs": [],
-      "source": [
-        "!pip install langtest"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Installing required dependencies"
+        "# Getting started with LangTest "
       ]
     },
     {
@@ -70,7 +48,7 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "!pip install \"langtest[evaluate,cohere,langchain]\" transformers==4.28.1"
+        "!pip install \"langtest[evaluate,cohere,langchain,transformers]\" "
       ]
     },
     {

diff --git a/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb b/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb
@@ -1,17 +1,5 @@
 {
   "cells": [
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "yR6kjOaiheKN"
-      },
-      "source": [
-        "# Harness and Its Parameters\n",
-        "\n",
-        "The Harness class is a testing class for Natural Language Processing (NLP) models. It evaluates the performance of a NLP model on a given task using test data and generates a report with test results.Harness can be imported from the LangTest library in the following way."
-      ]
-    },
     {
       "attachments": {},
       "cell_type": "markdown",
@@ -29,7 +17,7 @@
         "id": "3o5sAOfwL5qd"
       },
       "source": [
-        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Testing_Notebook.ipynb)"
+        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/HuggingFaceHub_QA_Summarization_Testing_Notebook.ipynb)"
       ]
     },
     {
@@ -51,7 +39,7 @@
         "id": "26qXWhCYhHAt"
       },
       "source": [
-        "# Getting started with LangTest on John Snow Labs"
+        "# Getting started with LangTest"
       ]
     },
     {
@@ -66,23 +54,7 @@
       },
       "outputs": [],
       "source": [
-        "!pip install \"langtest[evaluate,langchain]\" transformers==4.28.1"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "## Installing required dependencies"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "!pip install \"langtest[transformers,johnsnowlabs]\""
+        "!pip install \"langtest[evaluate,langchain,transformers]\" "
       ]
     },
     {