PAPA: Online Personalized Active Preference Alignment

This repository implements PAPA that efficiently finetunes a pretrained diffusion model for preference alignment without reward modeling. We provide distributed training and sampling codes for MNIST digits and fashion datasets. We build upon the DDPM code available here that mimics the huggingface stable diffusion unet architecture from scratch.

Overview of PAPA

Comparison of Samples Generated by Proposed PAPA and D3PO

Quickstart

• Create a new conda environment with python 3.8 then run below commands

• git clone https://github.com/NasikNafi/papa.git

• cd papa

• pip install -r requirements.txt Details of the requiremets.txt will be available soon.

• For training/sampling use the following commands with the desired configuration file.

• python -m tools.train_papa trains papa and saves generated images for evaluation

• python -m tools.sample_papa generates images for visualization

Configuration

• config/default_epapa.yaml - allows you to play with different components of DDPM and PAPA.

Output

Outputs will be saved according to the configuration present in yaml files.

For each run a folder named after sample_path key in config will be created under a directory of task_name key.

During training of PAPA the following outputs will be saved:

• Latest model checkpoint in task_name/sample_path directory.
• All generated samples that have been used to finetune the model at each interaction step.
• A folder named evaluation storing all generated images from every evaluation step.

During sampling the following output will be saved:

• Sampled image grid in task_name/sample_path.

Citations

@misc{sarkar2026papa,
      title={PAPA: Online Personalized Active Preference Alignment}, 
      author={Anindya Sarkar and Nasik Muhammad Nafi and Yevgeniy Vorobeychik},
      year={2026},
      eprint={####.#####},
      archivePrefix={arXiv},
      primaryClass={####}
}