DPMLBench

This repository contains the implementation of DPMLBench: Holistic Evaluation of Differentially Private Machine Learning.

Requirements

The environment can be set up using Anaconda with the following commands:

conda create --name dpml python=3.8
conda activate dpml
conda install pytorch=1.10.0 pytorch-cuda=11.7 torchvision -c pytorch -c nvidia
pip install -r requirements.txt

Introduction

This project uses sqlite to aid in storing model and experimental data. Both the member inference attack and the data analysis module interact through the database.

The project provides code for each algorithm to generate task scripts, named as gen_scripts.py in each algorithm folder. During the actual experiments, we used the gen_scripts.py to generate runing scripts and then utilized parallel tools to manage the training of these tasks.

DPMLBench/
    |- algorithms/   DPML algorithms implementation
    |- mem_inf/   Membership inference attack implementation
    |- models/   Code for Model architecture 
    |- origin_data/   Original statical data for replicate plots in the paper
    |- plot_scripts/   Generate plots and tables
    |- scripts/   Default folder for algorithms runing scripts generated by `gen_scripts.py`
    |- semisupervise/   Implement semisupervise learning
|- .gitignore
|- clip_only_utils.py   Implement only clipping experiments
|- data_factory.py    Split dataset and preprocess
|- db_merge.py   Database initialization
|- db_models.py   Database initialization
|- README.md  
|- replicate_figure.sh   Replocate plots
|- requirements.txt
|- sqlite_proxy.py   Code for database operation
|- tools.py   Tool codes

Replicate Plots

Run bash replicate_figure.sh to replicate plots in the paper.

Citation

@inproceedings{WZZCMLFC23,
    author = {Chengkun Wei and Minghu Zhao and Zhikun Zhang and Min Chen and Wenlong Meng and Bo Liu and Yuan Fan and Wenzhi Chen},
    title = {{DPMLBench: Holistic Evaluation of Differentially Private Machine Learning}},
    booktitle = {{ACM CCS}},
    publisher = {},
    year = {2023},
}

Acknowledgements

Our implementation refers to the source code from the following repositories:

• DPGEN
• PrivateSet
• RGP & GEP
• ALIBI
• PrivateKNN
• Handcraft DP
• MI Attack