-Supported data input formats are task-dependent. For `ner` and `text-classification`, the user is meant to provide a **`CoNLL`** or **`CSV`** dataset. For `question-answering` the user is meant to choose from a list of benchmark datasets.
+Supported data input formats are task-dependent. For `ner` and `text-classification`, the user is meant to provide a **`CoNLL`** or **`CSV`** dataset. For `question-answering`, `summarization` and `toxicity` the user is meant to choose from a list of benchmark datasets we support.
{:.table2}
| Task | Supported Data Inputs |
| - | - |
|**ner** |CoNLL and CSV|
-|**text-classification** |CSV
+|**text-classification** |CSV or a Dictionary (containing the name, subset, split, feature_column and target_column for loading the HF dataset.)
|**question-answering** |Select list of benchmark datasets
|**summarization** |Select list of benchmark datasets
+|**toxicity** |Select list of benchmark datasets
@@ -69,7 +70,7 @@ harness = Harness(task='ner',
### Text Classification
-There is 1 option for datasets to test Text Classification models: **`CSV`** datasets. Here are some details of what these may look like:
+There are 2 options for datasets to test Text Classification models: **`CSV`** datasets or a **`Dictionary`** containing the name, subset, split, feature_column and target_column for loading the HF datasets. Here are some details of what these may look like:
#### CSV Format for Text Classification
@@ -90,7 +91,7 @@ For `CSV` files, we support different variations of the column names. They are s
-#### Passing a Text Classification Dataset to the Harness
+#### Passing a CSV Text Classification Dataset to the Harness
In the Harness, we specify the data input in the following way:
@@ -107,6 +108,42 @@ harness = Harness(task='text-classification',
+#### Dictionary Format for Text Classification
+To handle text classification task for Hugging Face Datasets, the Harness class accepts the data parameter as a dictionary with following attributes:
+
+
+It's important to note that the default values for the **`split`**, **`feature_column`**, and **`target_column`** attributes are **`test`**, **`text`**, and **`label`**, respectively.
+
+```python
+{
+ "name": "",
+ "subset": "",
+ "feature_column": "",
+ "target_column": "",
+ "split": ""
+}
+```
+
+#### Passing a Hugging Face Dataset for Text Classification to the Harness
+
+In the Harness, we specify the data input in the following way:
+
+```python
+# Import Harness from the nlptest library
+from nlptest import Harness
+
+harness = Harness(task="text-classification", hub="huggingface",
+ model="distilbert-base-uncased-finetuned-sst-2-english",
+ data={"name":'glue',
+ "subset":"sst2",
+ "feature_column":"sentence",
+ "target_column":'label',
+ "split":"train"
+ })
+```
+
+
+
### Question Answering
To test Question Answering models, the user is meant to select a benchmark dataset from the following list:
diff --git a/docs/pages/docs/harness.md b/docs/pages/docs/harness.md
index de9c55284..69e278cb2 100644
--- a/docs/pages/docs/harness.md
+++ b/docs/pages/docs/harness.md
@@ -30,9 +30,9 @@ Here is a list of the different parameters that can be passed to the `Harness` c
| Parameter | Description |
| - | - |
|**task** |Task for which the model is to be evaluated ('text-classification', 'question-answering', 'ner')|
-|**model** |Pretrained pipeline or model from the corresponding hub, or path to a saved model from the corresponding hub, or PipelineModel object - see [Model Input](https://nlptest.org/docs/pages/docs/model_input) for more details
+|**model** |Pretrained pipeline or model from the corresponding hub, or path to a saved model from the corresponding hub, or PipelineModel object or a dictionary containing the names of the models you want to compare, each paired with its respective hub - see [Model Input](https://nlptest.org/docs/pages/docs/model_input) for more details
|**hub** |Hub (library) to use in back-end for loading model from public models hub or from path|
-|**data** |Path to the data to be used for evaluation. Should be `.csv` for text classification, or `.conll` or `.txt` file in CoNLL format for NER - see [Data Input](https://nlptest.org/docs/pages/docs/data_input) for more details
+|**data** |Path to the data to be used for evaluation. Should be `.csv` or a dictionary containing the name, subset, split, feature_column and target_column for loading the HF dataset for text classification, or `.conll` or `.txt` file in CoNLL format for NER - see [Data Input](https://nlptest.org/docs/pages/docs/data_input) for more details
|**config** |Path to the YAML file with configuration of tests to be performed
+
+## Custom Bias
+
+Supported Custom Bias Data Category:
+- `Country-Economic-Bias`
+- `Religion-Bias`
+- `Ethnicity-Name-Bias`
+- `Gender-Pronoun-Bias`
+
+#### How to Add Custom Bias
+
+To add custom bias, you can follow these steps:
+
+```python
+# Import Harness from the nlptest library
+from nlptest import Harness
+
+# Create a Harness object
+harness = Harness(
+ task="ner",
+ model='en_core_web_sm',
+ hub="spacy"
+)
+
+# Load custom bias data for country economic bias
+harness.pass_custom_bias_data(
+ file_path='economic_bias_data.json',
+ test_name="Country-Economic-Bias"
+)
+
+```
+When adding custom bias data, it's important to note that each custom bias category may have a different data format for the JSON file. Ensure that the JSON file adheres to the specific format required for each category.
+
+Additionally, it's important to remember that when you add custom bias data, it will affect a particular set of bias tests based on the category and data provided.
+
+To learn more about the data format and how to structure the JSON file for custom bias data, you can refer to the tutorial available [here](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/test-specific-notebooks/Custom_Bias_Demo.ipynb).
\ No newline at end of file
diff --git a/docs/pages/tutorials/tutorials.md b/docs/pages/tutorials/tutorials.md
index 0756fda8c..ee13cf20a 100644
--- a/docs/pages/tutorials/tutorials.md
+++ b/docs/pages/tutorials/tutorials.md
@@ -31,6 +31,7 @@ The following table gives an overview of the different tutorial notebooks. We ha
|Representation Tests |John Snow Labs |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/release%2F1.0.0/demo/tutorials/test-specific-notebooks/Representation_Demo.ipynb)|
|Robustness Tests |John Snow Labs |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/release%2F1.0.0/demo/tutorials/test-specific-notebooks/Robustness_DEMO.ipynb)|
|Toxicity Test |OpenAI |Toxicity|[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/llm_notebooks/Toxicity_NB.ipynb)|
+|Custom Bias |Spacy |NER/Text-Classification|[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/test-specific-notebooks/Custom_Bias_Demo.ipynb)|
|End-to-End Workflow |John Snow Labs |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/release%2F1.0.0/demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Notebook.ipynb)|
|End-to-End Custom Pipeline Workflow |John Snow Labs |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/release%2F1.0.0/demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Custom_Pipeline_Notebook.ipynb)|
|End-to-End Workflow |Spacy |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/release%2F1.0.0/demo/tutorials/end-to-end-notebooks/Spacy_Real_World_Notebook.ipynb)|
@@ -46,6 +47,10 @@ The following table gives an overview of the different tutorial notebooks. We ha
|TruthfulQA |OpenAI |Question-Answering |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/TruthfulQA_dataset.ipynb)|
|NarrativeQA |OpenAI |Question-Answering |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/NarrativeQA_Question_Answering.ipynb)|
|HellaSWag |OpenAI |Question-Answering |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/llm_notebooks/dataset-notebooks/HellaSwag_Question_Answering.ipynb)|
+|HuggingFaceDataset-Support |Hugging Face |Text-Classification |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/misc/HuggingFace_Dataset_Notebook.ipynb)|
+|Augmentation |Hugging Face |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/misc/Augmentation_Notebook.ipynb)|
+|Comparing Models |Hugging Face/John Snow Labs/Spacy |NER/Text-Classification |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/misc/Comparing_Models_Notebook.ipynb)|
+|Runtime Test |Hugging Face/John Snow Labs/Spacy |NER |[](https://colab.research.google.com/github/JohnSnowLabs/nlptest/blob/main/demo/tutorials/misc/RuntimeTest_Notebook.ipynb)|