DPImageBench: A Unified Benchmark for Differentially Private Image Synthesis Algorithms

1. Contents

• 1. Contents
• 2. Introduction
  • 2.1 Currently Supported Algorithms
• 3. Repo Contents
• 3. Get Start
  • 3.1 Installation
  • 3.2 Dataset and Files Preparation
  • 3.3 Training
  • 3.4 Inference
• 4. Contacts
• 5. Acknowledgment

2. Introduction

2.1 Currently Supported Algorithms

We list currently supported DP image synthesis methods as follows.

Methods	Link
DP-MERF	[AISTATS 2021] DP-MERF: Differentially Private Mean Embeddings With Randomfeatures for Practical Privacy-Preserving Data Generation
DP-Kernel	[NeuriPS 2023] Functional Renyi Differential Privacy for Generative Modeling
DPSDA	[ICLR 2024] Differentially Private Synthetic Data via Foundation Model {API}s 1: Images
G-PATE	[NeuriPS 2021] G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators
DataLens	[CCS 2021] DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
DP-GAN	[1802.06739] Differentially Private Generative Adversarial Network (arxiv.org)
DPDM	[TMLR 2023] Differentially Private Diffusion Models
PDP-Diffusion	[2302.13861] Differentially Private Diffusion Models Generate Useful Synthetic Images (arxiv.org)
DP-LDM	[2305.15759] Differentially Private Latent Diffusion Models (arxiv.org)
DP-Promise	[UESNIX Security 2024] DP-promise: Differentially Private Diffusion Probabilistic Models for Image Synthesis
PrivImage	[UESNIX Security 2024] PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining

3. Repo Contents

Below is the directory structure of the DPImageBench project, which encapsulates its three core functionalities within the watermark/, visualize/, and evaluation/ directories. To facilitate user understanding and demonstrate the toolkit's ease of use, we provide a variety of test cases. The test code can be found in the test/ directory.

DPImageBench/
├── config/                     # Configuration files for various watermark algorithms
│   ├── EWD.json           
│   ├── EXPEdit.json       
│   ├── EXP.json           
│   ├── KGW.json
│   ├── ITSEdit.json            
│   ├── SIR.json            
│   ├── SWEET.json         
│   ├── Unigram.json        
│   ├── UPV.json           
│   └── XSIR.json           
├── dataset/                    # Datasets used in the project
│   ├── c4/
│   ├── human_eval/
│   └── wmt16_de_en/
├── evaluation/                 # Evaluation module of MarkLLM, including tools and pipelines
│   ├── dataset.py              # Script for handling dataset operations within evaluations
│   ├── examples/               # Scripts for automated evaluations using pipelines
│   │   ├── assess_detectability.py  
│   │   ├── assess_quality.py    
│   │   └── assess_robustness.py   
│   ├── pipelines/              # Pipelines for structured evaluation processes
│   │   ├── detection.py    
│   │   └── quality_analysis.py 
│   └── tools/                  # Evaluation tools
│       ├── oracle.py
│       ├── success_rate_calculator.py  
        ├── text_editor.py         
│       └── text_quality_analyzer.py   
├── exceptions/                 # Custom exception definitions for error handling
│   └── exceptions.py
├── font/                       # Fonts needed for visualization purposes
├── MarkLLM_demo.ipynb          # Jupyter Notebook
├── test/                       # Test cases and examples for user testing
│   ├── test_method.py      
│   ├── test_pipeline.py    
│   └── test_visualize.py   
├── utils/                      # Helper classes and functions supporting various operations
│   ├── openai_utils.py     
│   ├── transformers_config.py 
│   └── utils.py            
├── visualize/                  # Visualization Solutions module of MarkLLM
│   ├── color_scheme.py    
│   ├── data_for_visualization.py  
│   ├── font_settings.py    
│   ├── legend_settings.py  
│   ├── page_layout_settings.py 
│   └── visualizer.py       
├── watermark/                  # Implementation framework for watermark algorithms
│   ├── auto_watermark.py       # AutoWatermark class
│   ├── base.py                 # Base classes and functions for watermarking
│   ├── ewd/                
│   ├── exp/               
│   ├── exp_edit/          
│   ├── kgw/
│   ├── its_edit/                 
│   ├── sir/               
│   ├── sweet/              
│   ├── unigram/           
│   ├── upv/                
│   └── xsir/               
├── README.md                   # Main project documentation
└── requirements.txt            # Dependencies required for the project

Folder Description

data contains tools for data loading.

DataLens and G-PATE are methods to be implemented.

dnnlib and torch_utils are used for load Inception V3.

models contains the implemented methods.

opacus is a modified opacus package.

exp contains my debug logs, and does not need to be uploaded.

Get Start on DPImageBench

conda activate dpbench
cd /p/fzv6enresearch/DPImageBench
python run.py configs/{model_name}/eps{epsilon}/{dataset_name}.yaml

Availabe model_name are [DataLens, DP-Kernel, DP-LDM, DP-MERF, DP-Promise, DPDM, DPSDA, G-PATE, PDP-Diffusion, PrivImage]. (DPSDA may face the issue of insufficient CPU memory.)

Availabe epsilon is [1.0].

Availabe dataset_name is [mnist].

So far, I have only implemented FID in our evaluation.