DOC Identify and Document Relevant Datasets from Safety Prompts

[nina-msft]

We recently discovered https://safetyprompts.com/, which has so many datasets!

We need help going through the website and creating a list of relevant datasets. A relevant dataset is one which contains red teaming prompts for different harm categories. For each relevant dataset, highlight which columns (e.g. prompt column in #420) can be used for red teaming prompts and post that information as a comment in this issue.

Expected Format (Example)

Name: LLM-LAT/harmful-dataset
Link: https://huggingface.co/datasets/LLM-LAT/harmful-dataset
Relevant Columns: "prompt"

Additional Context

We have datasets documented under orchestrators here: https://github.com/search?q=repo%3AAzure%2FPyRIT%20The%20dataset%20sources%20can%20be%20found%20at&type=code

Those dataset fetch functions are here: https://github.com/Azure/PyRIT/blob/main/pyrit/datasets/fetch_example_datasets.py

[divyaamin9825]

Hi! I'd like to work on this issue.

[Lakshmiaddepalli]

Hi! I'd like to work on this issue.

[divyaamin9825]

Name: jailbreak_llms/data/forbidden_question
Link: https://github.com/verazuo/jailbreak_llms/blob/main/data/forbidden_question/forbidden_question_set.csv
Relevant Columns: "content_policy_name","question"

[divyaamin9825]

Name: LibrAI/do-not-answer
Link: https://huggingface.co/datasets/LibrAI/do-not-answer
Relevant Columns: "risk_area","types_of_harm","specific_harms","question"

[divyaamin9825]

Name: McGill-NLP/stereoset
Link: https://huggingface.co/datasets/McGill-NLP/stereoset
Relevant columns: "target", "bias_type", "context", "sentences"