The AVID (AI Vulnerability Database) team is examining a few large language models (LLMs) on Hugging Face. We will develop a way to evaluate and catalog their vulnerabilities in the hopes of encouraging the community to contribute. As a first step, we’re going to pick a single model and try to evaluate it for vulnerabilities on a specific task. Once we have done one model, we’ll see if we can generalize our data sets and tools to function broadly on the Hugging Face platform.
Build a foundation for evaluating LLMs using the Hugging Face platform and start populating our database with real incidents.
- Build, test, and refine our own data sets for evaluating models
- Identify existing data sets we want to use for evaluating models (Ex. Stereoset, wino_bias, etc.)
- Test different tools and methods for evaluating LLMs so we can start to create and support some for cataloging vulnerabilities in our database
- Start populating the database with known, verified, and discovered vulnerabilities for models hosted on Hugging Face
The links below should help anyone who wants to support the project find a place to start. They are not exhaustive, and people should feel free to add anything relevant.
- Huggingface.co - platform for hosting data sets, models, etc.
- Papers With Code - a platform for the ML community to share research, it may have additional data sets or papers
- Potential Models
- Data Sets
- StereoSet - StereoSet is a dataset that measures stereotype bias in language models. StereoSet consists of 17,000 sentences that measure model preferences across gender, race, religion, and profession.
- Wino_bias - WinoBias, a Winograd-schema dataset for coreference resolution focused on gender bias.
- Jigsaw_unintended_bias - The main target for this dataset is toxicity prediction. Several toxicity subtypes are also available, so the dataset can be used for multi-attribute prediction.
- BigScienceBiasEval/bias-shades - This dataset was curated by hand-crafting stereotype sentences by native speakers from the culture which is being targeted. (Seems incomplete)
- md_gender_bias - The dataset can be used to train a model for classification of various kinds of gender bias.
- social_bias_frames - This dataset supports both classification and generation. Sap et al. developed several models using the SBIC.
- BIG-bench/keywords_to_tasks.md at main - includes many options for testing bias of different types (gender, religion, etc.)
- FB Fairscore - Has a wide selection of sources, focuses on gender (including non-binary).
- Papers
- Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurement
- On the Dangers of Stochastic Parrots
- Language (Technology) is Power: A Critical Survey of “Bias” in NLP
- Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models
- Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies