ReLay: Personalized LLM-Generated Plain-Language Summaries for Better Understanding of Health Information, but at What Cost?

ReLay generates and evaluates personalized plain language summaries (PLS) of scientific abstracts. Summaries are adapted to individual readers using a range of personalization strategies, from zero-shot prompting to retrieval-augmented generation (RAG).

---

Project Structure

ReLay/
├── Datasets/
│   ├── participant.json              # Participant interaction data (anonymized, IDs 1-50)
│   ├── cochrane_abstracts.csv        # All Cochrane systematic review abstracts
│   ├── participant_surveys.csv       # Survey responses for 50 study participants
│   ├── health_surveys.csv            # Full survey pool (128 respondents)
│   └── articles.py                   # Reference article texts and site_key_map for style/readability
├── Experiment/
│   ├── Non-personalized/
│   │   └── non_personalized.py       # Generate non-personalized PLS (baseline)
│   └── Personalized/
│       ├── Profile-Based Prompting/
│       │   ├── backstory.py          # Generate backstory-personalized PLS
│       │   └── metadata.py           # Generate metadata-personalized PLS
│       └── Retrieval-Based Prompting/
│           ├── within_user_rag.py    # Generate within-user RAG PLS
│           └── cross_user_rag.py     # Generate cross-user RAG PLS
├── Metrics/
│   ├── Personalization/
│   │   ├── knowledge.py             # Knowledge alignment (LLM judge)
│   │   ├── readability.py           # Readability error (grade-level diff)
│   │   └── style.py                 # Style alignment (19-dim feature vector)
│   └── Safety/
│       ├── bias_reinforcement.py    # Bias-reinforcing framing (LLM judge)
│       └── hallucination.py         # Faithfulness and factuality (claim-level)
├── Experiments/                       # Downloaded from HuggingFace (see Setup)
│   ├── pls_results.json             # All personalized PLS results
│   └── nonpersonalized_results.json # Nonpersonalized baseline results
├── outputs/                          # Aggregated evaluation summary CSVs
├── download_data.py                  # Download experiment data from HuggingFace
├── requirements.txt
└── README.md

---

Setup

Requirements

pip install -r requirements.txt
python -m spacy download en_core_web_sm
python download_data.py

Note: Generated PLS results are hosted on Hugging Face. Run python download_data.py after cloning to download them into Experiments/.

Environment

Set your OpenAI API key before running any script that calls the OpenAI API:

export OPENAI_API_KEY=your_key_here

---

Data

File	Description
Datasets/participant.json	Anonymized participant data (IDs 1-50) with demographics, per-abstract interaction logs, readability baselines, and style vectors
Datasets/cochrane_abstracts.csv	Source abstracts from Cochrane systematic reviews, with comprehension questions
Datasets/participant_surveys.csv	Health literacy, AI usage, and information-seeking survey for the 50 study participants
Datasets/health_surveys.csv	Full survey pool (128 respondents; IDs 1-50 are study participants, 51-128 are survey-only)
Datasets/articles.py	Reference article texts from health websites participants reported reading (used to derive per-user readability baselines and style vectors)

participant.json Schema

Each entry represents one participant (50 total):

Field	Description
prolific_id	Anonymized integer ID (1-50)
readability_score	User's baseline reading grade level (avg of Flesch-Kincaid, Gunning Fog, SMOG, Coleman-Liau over their reported reading material)
style_vector	19-dimensional style feature vector derived from the user's reported reading material
csv_info	Demographics and survey metadata (age, sex, ethnicity, education, health literacy, AI usage, etc.)
phases	Nested study interaction data, organized by phase and batch

Study phases:

• static (batches 1, 2) — participants read abstracts, rated term familiarity, requested support (Definition/Background/Example), and answered comprehension questions
• interactive (batch 3) — participants interacted with a chatbot to understand abstracts

Note: The interactive phase conversation logs (user questions asked during the chatbot interaction) are not included in this release. They will be released at a later date as we are still refining them.

Per-abstract fields (within each batch):

• abstract_title, abstract, human_written_pls — source materials
• question_1–question_5 — comprehension questions with answer choices and correct answers
• term_familarity — list of terms with familiarity scores and requested support types
• sata — participant's select-all-that-apply comprehension answers
• likert — participant's Likert-scale quality ratings

cochrane_abstracts.csv Schema

Column	Description
abstract_id	Unique identifier
topic	Health topic (e.g., Diabetes, Cancer)
method	Treatment type (e.g., Management, Prevention)
link	Cochrane source URL
abstract	Full systematic review abstract
human_written_pls	Cochrane's human-written plain-language summary
question_1–question_5	Comprehension questions
question_N_answers_choices	Answer options for each question
question_N_correct_answers	Correct answers
terms_chosen	Key terms selected for the term familiarity task

articles.py Schema

Contains two objects:

• articles — dictionary mapping 15 site keys (e.g., "webmd", "mayoclinic", "cdc") to reference article text scraped from those sites
• site_key_map — dictionary mapping 35 user-reported site name variants (e.g., "Mayo Clinic", "mayoclinic.org") to canonical site keys

These are used to compute each user's readability_score and style_vector by analyzing the text of articles from sites they reported reading.

Survey CSVs Schema

Both participant_surveys.csv (50 participants) and health_surveys.csv (128 respondents) share the same columns:

Column group	Examples
participant_id	Anonymized integer ID
Demographics	Employment_Status, Income, Hispanic
Health topic familiarity	familiar_Cancer, familiar_Diabetes, etc. (Likert scale)
Health topic interest	interested_Cancer, interested_Diabetes, etc.
Preferred info method	method_Cancer, method_Diabetes, etc.
AI usage	Tool frequency, tools used, types of questions asked
Health literacy	Numeracy question, confidence in finding health info, verification habits
Info-seeking behavior	Websites used, reliability ratings of sources

---

Generation

All generation scripts follow the same CLI pattern:

--data         Path to participant JSON (default: Datasets/participant.json)
--models       One or more model aliases (required)
--backend      openai | hf | all  (default: all, auto-routes each model)
--output-dir   Directory for output CSVs (default: outputs/)

Model Aliases

Alias	Model
gpt4o	gpt-4o
gpt5_2	gpt-5.2
mistral_7b	mistralai/Mistral-7B-Instruct-v0.3
medgemma_27b	google/medgemma-27b-it
qwen3_4b	Qwen/Qwen3-4B-Instruct-2507

Non-Personalized (Baseline)

python Experiment/Non-personalized/non_personalized.py --models gpt4o gpt5_2 --backend openai
python Experiment/Non-personalized/non_personalized.py --models mistral_7b medgemma_27b qwen3_4b --backend hf

Backstory-Personalized

python "Experiment/Personalized/Profile-Based Prompting/backstory.py" --models gpt4o gpt5_2 --backend openai

Metadata-Personalized

python "Experiment/Personalized/Profile-Based Prompting/metadata.py" --models gpt4o gpt5_2 --backend openai

Within-User RAG

python "Experiment/Personalized/Retrieval-Based Prompting/within_user_rag.py" --models gpt4o --n-priors 1
python "Experiment/Personalized/Retrieval-Based Prompting/within_user_rag.py" --models gpt4o --n-priors 2

Cross-User RAG

# Step 1: preprocess (builds cross-user KNN matches, run once)
python "Experiment/Personalized/Retrieval-Based Prompting/cross_user_rag.py" --mode preprocess

# Step 2: generate
python "Experiment/Personalized/Retrieval-Based Prompting/cross_user_rag.py" --mode generate --models gpt4o --n-priors 1

---

Evaluation

All evaluation scripts read from Experiments/pls_results.json and write results in-place. Run from the repo root.

Personalization Metrics

Metric	Script	Description
Knowledge alignment	Metrics/Personalization/knowledge.py	LLM judge scores whether PLS fulfills user's information needs (static: term support, interactive: user questions). Score in [0, 1].
Readability error	Metrics/Personalization/readability.py	Absolute difference between PLS reading grade level and user baseline. Lower is better.
Style alignment	Metrics/Personalization/style.py	Closeness between PLS and user's reading style across 19 features (lexical, vocabulary richness, emotion). Score in [0, 1].

Safety Metrics

Metric	Script	Description
Bias reinforcement	Metrics/Safety/bias_reinforcement.py	LLM judge flags PLS that introduce, amplify, or normalize harmful biased framing. Binary 0/1.
Hallucination	Metrics/Safety/hallucination.py	Claim-level evaluation: atomize PLS into claims, judge simplifications against abstract (faithfulness) and expansions against PubMed evidence via MedRAG (factuality).

Running Evaluations

# Knowledge alignment (requires OpenAI API)
python Metrics/Personalization/knowledge.py

# Readability error (local, no API needed)
python Metrics/Personalization/readability.py

# Style alignment (local, needs GPU for emotion model)
python Metrics/Personalization/style.py

# Bias evaluation (requires OpenAI API)
python Metrics/Safety/bias_reinforcement.py

# Hallucination evaluation (requires OpenAI API + MedRAG corpus)
python Metrics/Safety/hallucination.py

Evaluation Models

Component	Model
Knowledge judge	gpt-5.2
Bias judge	gpt-5.2
Claim atomization	gpt-4o-mini
Hallucination judge	gpt-5.2-mini
Evidence retrieval	MedCPT over PubMed (k=5)
Style emotion features	SamLowe/roberta-base-go_emotions

---

Reproducing Results

1. Clone and install:

   git clone https://github.com/jchan58/ReLay.git
   cd ReLay
   pip install -r requirements.txt
   python -m spacy download en_core_web_sm
   python download_data.py

2. Generate PLS (or use pre-generated results in Experiments/):

   export OPENAI_API_KEY=your_key_here
   python Experiment/Non-personalized/non_personalized.py --models gpt4o

3. Run evaluations:

   python Metrics/Personalization/readability.py
   python Metrics/Personalization/style.py
   python Metrics/Personalization/knowledge.py
   python Metrics/Safety/bias_reinforcement.py
   python Metrics/Safety/hallucination.py

Pre-computed evaluation results are available in outputs/:

• hallucination_personalized_summary.csv — faithfulness/factuality by model and method
• hallucination_personalized_by_model_method.csv — detailed averages per model-method pair
• hallucination_nonpersonalized_by_model.csv — nonpersonalized hallucination by model
• hallucination_nonpersonalized_by_model_condition.csv — nonpersonalized by model and study condition

---

Method Names in Data

The method field in Experiments/pls_results.json maps to the generation strategies as follows:

Method in data	Strategy	Script
zeroshot prompt / zero-shot prompt	Non-personalized baseline	non_personalized.py
cot prompt	Chain-of-thought baseline	non_personalized.py
zeroshot backstory prompt / zero-shot backstory prompt / cot backstory prompt	Backstory-personalized	backstory.py
zero-shot metadata prompt	Metadata-personalized	metadata.py
top_similar_personalized_prompt_no_prior_human_pls	Within-user RAG (top-1)	within_user_rag.py
top2_similar_personalized_prompt_no_prior_human_pls	Within-user RAG (top-2)	within_user_rag.py
top1_cross_user_personalized_prompt_no_prior_human_pls	Cross-user RAG (top-1)	cross_user_rag.py
top2_cross_user_personalized_prompt_no_prior_human_pls	Cross-user RAG (top-2)	cross_user_rag.py

---

Notes

• All evaluation scripts skip already-scored records and support resuming from partial runs.
• HuggingFace models are loaded one at a time and unloaded from GPU between runs to manage memory.
• Hallucination evaluation requires a local MedRAG PubMed corpus (see MedRAG for setup).
• Participant IDs are anonymized integers (1–50). The survey CSVs use the column name participant_id; all other files use prolific_id. They refer to the same anonymized IDs.
• Model names in the data may vary in format (e.g., gpt-4o / gpt4o, mistral 7b / mistral_7b) due to different generation runs. All refer to the same models listed in the Model Aliases table.
• The field term_familarity in participant.json is an intentional spelling preserved from the original data collection system.

---

Citation

@article{chan2026relay,
  title={ReLay: Personalized LLM-Generated Plain-Language Summaries for Better Understanding of Health Information, but at What Cost?},
  author={Chan, Joey and Han, Yikun and Chen, Jingyuan and Fang, Samuel and Gryboski, Lauren D and Lee, Alexandra and Tanna, Sheel and Zhu, Qingqing and Lu, Zhiyong and Wang, Lucy Lu and others},
  journal={arXiv preprint arXiv:2605.00468},
  year={2026}
}