QueerGen: Evaluating Representational Harm Toward Queer Identities in Language Models

QueerGen is a framework for evaluating how large language models (LLMs) represent queer and non-normative identities. It uses controlled prompt templates and automated analysis to assess sentiment, regard, toxicity, and lexical diversity across completions generated by different models for unmarked, non-queer-marked, and queer-marked subjects.

---

📁 Project Structure

.
├── dataset/
│   ├── templates.csv
│   ├── subjects.csv
│   ├── markers.csv
│   └── template_complete.csv        # Auto-generated prompts
│
├── generations/                     # Generated completions (by model & prompt)
├── evaluations/                     # Sentiment, regard, toxicity, diversity scores
├── graphs/                          # Generated charts
├── tables/                          # CSV tables used in paper
│
├── env_config.sh                    # Environment + model setup
├── main.py                          # Runs full pipeline
├── graphs.ipynb                     # Regenerate plots and tables (Jupyter)
│
├── src/
│   ├── template.py                  # Prompt creation
│   ├── models.py                    # Completion generation (API + local)
│   ├── evaluation.py                # Evaluation metrics
│   └── lib.py                       # Utilities
└── .env                             # API keys (required)

---

⚙️ Installation

Install all required libraries and download models by running:

bash env_config.sh

This script will:

• Install Python dependencies
• Download local Ollama models (e.g., LLaMA 3, Gemma, DeepSeek)
• Set up necessary folder structure
• Clone regard evaluation models

---

▶️ Run the Full Pipeline

To run all steps — from prompt generation to model completion and evaluation:

python main.py

This will:

• Create all templates (template_complete.csv)
• Generate sentence completions for all models and prompts
• Evaluate completions for sentiment, regard, toxicity, and diversity
• Save all outputs to the appropriate folders

---

📊 Plotting & Table Generation

To generate or update all the visualizations and tables used in the analysis, open:

graphs.ipynb

This Jupyter notebook reads evaluation data and generates:

• All charts in graphs/
• All result tables in tables/

---

🔐 API Configuration

The .env file must contain the required API keys for the external models and services:

OPENAI_API_KEY=your_openai_key
GENAI_API_KEY=your_genai_key
DEEPSEEK_API_KEY=your_deepseek_key
PERSPECTIVE_API_KEY=your_perspective_key

Key	Used For
OPENAI_API_KEY	GPT-4, GPT-4o-mini
GENAI_API_KEY	Gemini 2.0 Flash
DEEPSEEK_API_KEY	DeepSeek R1
PERSPECTIVE_API_KEY	Toxicity evaluation

---

🧠 Script Functionality Overview

• src/models.py: Handles prompt completion using both open- and closed-access LLMs via APIs or Ollama. Each model is wrapped to ensure consistent generation and output format.

• src/evaluation.py: Computes automatic fairness metrics for all generated sentences, including:

  • Sentiment (VADER)
  • Regard (via pre-trained classifier)
  • Toxicity (Perspective API)
  • Lexical Diversity (type-token ratio)

---

License: Academic research only. Please cite the corresponding paper if you use QueerGen in your work.