release: private evolution for images using simulators

Author: fjxmlzn

README.md

@@ -8,6 +8,9 @@ This repo is a Python library to **generate differentially private (DP) syntheti
 * **Differentially Private Synthetic Data via Foundation Model APIs 2: Text**  
     [[paper (ICML 2024 Spotlight)]](https://proceedings.mlr.press/v235/xie24g.html) [[paper (arxiv)](https://arxiv.org/abs/2403.01749)] [[website](https://alphapav.github.io/augpe-dpapitext)]  
     **Authors:** [[Chulin Xie](https://alphapav.github.io/)], [[Zinan Lin](https://zinanlin.me/)], [[Arturs Backurs](https://www.mit.edu/~backurs/)], [[Sivakanth Gopi](https://www.microsoft.com/en-us/research/people/sigopi/)], [[Da Yu](https://dayu11.github.io/)], [[Huseyin Inan](https://www.microsoft.com/en-us/research/people/huinan/)], [[Harsha Nori](https://www.microsoft.com/en-us/research/people/hanori/)], [[Haotian Jiang](https://jhtdavid96.wixsite.com/jianghaotian)], [[Huishuai Zhang](https://huishuai-git.github.io/)], [[Yin Tat Lee](https://yintat.com/)], [[Bo Li](https://aisecure.github.io/)], [[Sergey Yekhanin](http://www.yekhanin.org/)]
+* **Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Models**  
+    [[paper (arxiv)](https://arxiv.org/abs/2502.05505)]
+    **Authors:** [[Zinan Lin](https://zinanlin.me/)], [[Tadas Baltrusaitis](https://www.microsoft.com/en-us/research/people/tabaltru/)], [[Sergey Yekhanin](http://www.yekhanin.org/)]
 
 Please refer to [this repo](https://github.com/fjxmlzn/private-evolution-papers) for the full list of Private Evolution papers and code repositories related to PE.
 
@@ -16,8 +19,9 @@ Please refer to the [documentation](https://microsoft.github.io/DPSDA/) for more
 
 ## News
 
-* `1/8/2025`: **Text generation** based on the paper [`Differentially Private Synthetic Data via Foundation Model APIs 2: Text`](https://arxiv.org/abs/2403.01749) has been integrated into the library! If you want to reproduce the results in the [paper](https://arxiv.org/abs/2403.01749), please refer to [our original codebase](https://github.com/AI-secure/aug-pe).
-* `11/21/2024`: The refactored codebase for **image generation** based on the paper [`Differentially Private Synthetic Data via Foundation Model APIs 1: Images`](https://arxiv.org/abs/2305.15560) has been released! It is completely refactored to be more modular and easier to use and extend. The code originally published with the [paper](https://arxiv.org/abs/2305.15560) has been moved to the [deprecated](https://github.com/microsoft/DPSDA/tree/deprecated) branch in this repository, which is no longer maintained.
+* `2/11/2025`: **Image generation with simulator APIs** based on the paper [`Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Models`](https://arxiv.org/abs/2502.05505) has been released in this library!
+* `1/8/2025`: **Text generation with foundation model APIs** based on the paper [`Differentially Private Synthetic Data via Foundation Model APIs 2: Text`](https://arxiv.org/abs/2403.01749) has been integrated into the library! If you want to reproduce the results in the [paper](https://arxiv.org/abs/2403.01749), please refer to [our original codebase](https://github.com/AI-secure/aug-pe).
+* `11/21/2024`: The refactored codebase for **image generation with foundation model APIs** based on the paper [`Differentially Private Synthetic Data via Foundation Model APIs 1: Images`](https://arxiv.org/abs/2305.15560) has been released! It is completely refactored to be more modular and easier to use and extend. The code originally published with the [paper](https://arxiv.org/abs/2305.15560) has been moved to the [deprecated](https://github.com/microsoft/DPSDA/tree/deprecated) branch in this repository, which is no longer maintained.
 
 ## Contributing
 

doc/build_autodoc.sh

@@ -1,2 +1,2 @@
-sphinx-apidoc -e -f --module-first -d 7 -o source/api ../pe ../pe/*/test* ../pe/*/bk*
+sphinx-apidoc -e -f --module-first -d 7 -o source/api ../pe ../pe/*/test* ../pe/*/*/test* ../pe/*/bk*
 make clean html

doc/source/api/pe.api.image.avatar_api.rst

@@ -0,0 +1,7 @@
+pe.api.image.avatar\_api module
+===============================
+
+.. automodule:: pe.api.image.avatar_api
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.api.image.draw_text_api.rst

@@ -0,0 +1,7 @@
+pe.api.image.draw\_text\_api module
+===================================
+
+.. automodule:: pe.api.image.draw_text_api
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.api.image.nearest_image_api.rst

@@ -0,0 +1,7 @@
+pe.api.image.nearest\_image\_api module
+=======================================
+
+.. automodule:: pe.api.image.nearest_image_api
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.api.image.rst

@@ -20,5 +20,8 @@ Submodules
 .. toctree::
    :maxdepth: 7
 
+   pe.api.image.avatar_api
+   pe.api.image.draw_text_api
    pe.api.image.improved_diffusion_api
+   pe.api.image.nearest_image_api
    pe.api.image.stable_diffusion_api

doc/source/api/pe.callback.common.compute_precision_recall.rst

@@ -0,0 +1,7 @@
+pe.callback.common.compute\_precision\_recall module
+====================================================
+
+.. automodule:: pe.callback.common.compute_precision_recall
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.common.rst

@@ -13,4 +13,5 @@ Submodules
    :maxdepth: 7
 
    pe.callback.common.compute_fid
+   pe.callback.common.compute_precision_recall
    pe.callback.common.save_checkpoints

doc/source/api/pe.callback.image.dpimagebench_classify_images.rst

@@ -0,0 +1,7 @@
+pe.callback.image.dpimagebench\_classify\_images module
+=======================================================
+
+.. automodule:: pe.callback.image.dpimagebench_classify_images
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.image.dpimagebench_lib.densenet.rst

@@ -0,0 +1,7 @@
+pe.callback.image.dpimagebench\_lib.densenet module
+===================================================
+
+.. automodule:: pe.callback.image.dpimagebench_lib.densenet
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.image.dpimagebench_lib.ema.rst

@@ -0,0 +1,7 @@
+pe.callback.image.dpimagebench\_lib.ema module
+==============================================
+
+.. automodule:: pe.callback.image.dpimagebench_lib.ema
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.image.dpimagebench_lib.resnet.rst

@@ -0,0 +1,7 @@
+pe.callback.image.dpimagebench\_lib.resnet module
+=================================================
+
+.. automodule:: pe.callback.image.dpimagebench_lib.resnet
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.image.dpimagebench_lib.resnext.rst

@@ -0,0 +1,7 @@
+pe.callback.image.dpimagebench\_lib.resnext module
+==================================================
+
+.. automodule:: pe.callback.image.dpimagebench_lib.resnext
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.image.dpimagebench_lib.rst

@@ -0,0 +1,19 @@
+pe.callback.image.dpimagebench\_lib package
+===========================================
+
+.. automodule:: pe.callback.image.dpimagebench_lib
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Submodules
+----------
+
+.. toctree::
+   :maxdepth: 7
+
+   pe.callback.image.dpimagebench_lib.densenet
+   pe.callback.image.dpimagebench_lib.ema
+   pe.callback.image.dpimagebench_lib.resnet
+   pe.callback.image.dpimagebench_lib.resnext
+   pe.callback.image.dpimagebench_lib.wrn

doc/source/api/pe.callback.image.dpimagebench_lib.wrn.rst

@@ -0,0 +1,7 @@
+pe.callback.image.dpimagebench\_lib.wrn module
+==============================================
+
+.. automodule:: pe.callback.image.dpimagebench_lib.wrn
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.callback.image.rst

@@ -6,11 +6,20 @@ pe.callback.image package
    :undoc-members:
    :show-inheritance:
 
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 7
+
+   pe.callback.image.dpimagebench_lib
+
 Submodules
 ----------
 
 .. toctree::
    :maxdepth: 7
 
+   pe.callback.image.dpimagebench_classify_images
    pe.callback.image.sample_images
    pe.callback.image.save_all_images

doc/source/api/pe.data.image.celeba.rst

@@ -0,0 +1,7 @@
+pe.data.image.celeba module
+===========================
+
+.. automodule:: pe.data.image.celeba
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.data.image.digiface1m.rst

@@ -0,0 +1,7 @@
+pe.data.image.digiface1m module
+===============================
+
+.. automodule:: pe.data.image.digiface1m
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.data.image.mnist.rst

@@ -0,0 +1,7 @@
+pe.data.image.mnist module
+==========================
+
+.. automodule:: pe.data.image.mnist
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.data.image.rst

@@ -14,5 +14,8 @@ Submodules
 
    pe.data.image.camelyon17
    pe.data.image.cat
+   pe.data.image.celeba
    pe.data.image.cifar10
+   pe.data.image.digiface1m
    pe.data.image.image
+   pe.data.image.mnist

doc/source/api/pe.embedding.image.fld_inception.rst

@@ -0,0 +1,7 @@
+pe.embedding.image.fld\_inception module
+========================================
+
+.. automodule:: pe.embedding.image.fld_inception
+   :members:
+   :undoc-members:
+   :show-inheritance:

doc/source/api/pe.embedding.image.rst

@@ -12,4 +12,5 @@ Submodules
 .. toctree::
    :maxdepth: 7
 
+   pe.embedding.image.fld_inception
    pe.embedding.image.inception

doc/source/conf.py

@@ -69,4 +69,7 @@
     ("py:class", "improved_diffusion.respace.SpacedDiffusion"),
     ("py:class", "improved_diffusion.unet.UNetModel"),
     ("py:class", "omegaconf.dictconfig.DictConfig"),
+    ("py:class", "python_avatars.Avatar"),
+    ("py:class", "torch.utils.data.DataLoader"),
+    ("py:class", "torch.nn.Module"),
 ]

doc/source/getting_started/examples.rst

@@ -10,7 +10,13 @@ Images
 
     * **CIFAR10 dataset**: `This example <CIFAR10 example_>`__ shows how to generate differentially private synthetic images for the `CIFAR10 dataset`_ using the APIs from a pre-trained `ImageNet diffusion model`_.
     * **Camelyon17 dataset**: `This example <Camelyon17 example_>`__ shows how to generate differentially private synthetic images for the `Camelyon17 dataset`_ using the APIs from a pre-trained `ImageNet diffusion model`_.
-    * **Cat dataset**: `This example <Cat example_>`__ shows how to generate differentially private synthetic images of the `Cat dataset`_ using the APIs from `Stable Diffusion`_.
+    * **Cat dataset**: `This example <Cat example_>`__ shows how to generate differentially private synthetic images for the `Cat dataset`_ using the APIs from `Stable Diffusion`_.
+
+* Using **simulators** as the APIs. These examples follow the experimental settings in the paper `Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Models <pe3_paper_>`__.
+
+    * **MNIST dataset**: `This example <MNIST example_>`__ shows how to generate differentially private synthetic images for the `MNIST dataset`_ using a text render.
+    * **CelebA dataset (simulator-generated data)**: `This example <CelebA DigiFace1M example_>`__ shows how to generate differentially private synthetic images for the `CelebA dataset`_ using `the generated data from a computer graphics-based renderer for face images <DigiFace1M_>`__.
+    * **CelebA dataset (weak simulator)**: `This example <CelebA avatar example_>`__ shows how to generate differentially private synthetic images for the `CelebA dataset`_ using `a rule-based avatar generator <python_avatar_>`__.
 
 Text
 ----
@@ -36,22 +42,36 @@ These examples follow the experimental settings in the paper `Differentially Pri
 .. _ImageNet diffusion model: https://github.com/openai/improved-diffusion
 .. _Stable Diffusion: https://huggingface.co/CompVis/stable-diffusion-v1-4
 
+.. _DigiFace1M: https://github.com/microsoft/DigiFace1M
+.. _python_avatar: https://github.com/ibonn/python_avatars
+
 .. _Cat dataset: https://www.kaggle.com/datasets/fjxmlzn/cat-cookie-doudou
 .. _CIFAR10 dataset: https://www.cs.toronto.edu/~kriz/cifar.html
 .. _Camelyon17 dataset: https://camelyon17.grand-challenge.org/
+
+.. _MNIST dataset: https://pytorch.org/vision/main/generated/torchvision.datasets.MNIST.html
+.. _CelebA dataset: https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
+
 .. _Yelp dataset: https://github.com/AI-secure/aug-pe/tree/main/data
 .. _OpenReview dataset: https://github.com/AI-secure/aug-pe/tree/main/data
 .. _PubMed dataset: https://github.com/AI-secure/aug-pe/tree/main/data
 
 .. _CIFAR10 example: https://github.com/microsoft/DPSDA/blob/main/example/image/diffusion_model/cifar10_improved_diffusion.py
 .. _Camelyon17 example: https://github.com/microsoft/DPSDA/blob/main/example/image/diffusion_model/camelyon17_improved_diffusion.py
 .. _Cat example: https://github.com/microsoft/DPSDA/blob/main/example/image/diffusion_model/cat_stable_diffusion.py
+
+.. _MNIST example: https://github.com/microsoft/DPSDA/blob/main/example/image/simulator/mnist_text_render.py
+.. _CelebA DigiFace1M example: https://github.com/microsoft/DPSDA/blob/main/example/image/simulator/celeba_digiface1m.py
+.. _CelebA avatar example: https://github.com/microsoft/DPSDA/blob/main/example/image/simulator/celeba_avatar.py
+
 .. _Yelp OpenAI example: https://github.com/microsoft/DPSDA/blob/main/example/text/yelp_openai/main.py
 .. _Yelp Huggingface example: https://github.com/microsoft/DPSDA/blob/main/example/text/yelp_huggingface/main.py
 .. _Openreview OpenAI example: https://github.com/microsoft/DPSDA/blob/main/example/text/openreview_openai/main.py
 .. _Openreview Huggingface example: https://github.com/microsoft/DPSDA/blob/main/example/text/openreview_huggingface/main.py
 .. _PubMed OpenAI example: https://github.com/microsoft/DPSDA/blob/main/example/text/pubmed_openai/main.py
 .. _PubMed Huggingface example: https://github.com/microsoft/DPSDA/blob/main/example/text/pubmed_huggingface/main.py
 
+
 .. _pe1_paper: https://arxiv.org/abs/2305.15560
-.. _pe2_paper: https://arxiv.org/abs/2403.01749
+.. _pe2_paper: https://arxiv.org/abs/2403.01749
+.. _pe3_paper: https://arxiv.org/abs/2502.05505

doc/source/getting_started/intro.rst

@@ -14,15 +14,15 @@ Key Features
 
 Compared to other DP synthetic data alternatives, **Private Evolution** has the following key features:
 
-* ✅ **No training needed!** **Private Evolution** only requires the inference APIs of foundation models. Therefore, it can leverage any state-of-the-art black-box models (e.g., GPT-4) and open-source models (e.g., Stable Diffusion, Llama).
+* ✅ **No training needed!** **Private Evolution** only requires the inference APIs of foundation models or non-neural-network data synthesis tools. Therefore, it can leverage any state-of-the-art black-box models (e.g., GPT-4), open-source models (e.g., Stable Diffusion, Llama), or tools (e.g., computer graphics-based image synthesis tools).
 * ✅ **Protects privacy even from the API provider.** Even when using APIs from a third-party provider, you can rest assured that the information of individuals in the original dataset is still protected, as all API queries made from **Private Evolution** are also differentially private.
 * ✅ **Works across images, text, etc.** **Private Evolution** can generate synthetic data for various data types, including images and text. More data modalities are coming soon!
 * ✅ **Could even match/beat SoTA training-based methods in data quality.** **Private Evolution** can generate synthetic data that is statistically similar to the original data, and in some cases, it can even match or beat the state-of-the-art training-based methods in data quality even though it does not require any training.
 
 What This Library Provides
 --------------------------
 
-**This library is the official Python package of Private Evolution**. It allows you to generate differentially private synthetic data (e.g., images, text) using the **Private Evolution** algorithm. This library is designed to be easy to use, flexible, modular, and extensible. It provides several popular foundation model APIs, and you can easily extend it to work with your own foundation models (and/or APIs), data types, or new **Private Evolution** algorithms if needed.
+**This library is the official Python package of Private Evolution**. It allows you to generate differentially private synthetic data (e.g., images, text) using the **Private Evolution** algorithm. This library is designed to be easy to use, flexible, modular, and extensible. It provides several popular foundation models and data synthesis tools, and you can easily extend it to work with your own foundation models (and/or APIs), data synthesis tools, data types, or new **Private Evolution** algorithms if needed.
 
 The source code of this **Private Evolution** library is available at https://github.com/microsoft/DPSDA.
 
@@ -37,4 +37,7 @@ If you use **Private Evolution** in your research or work, please cite the follo
 .. literalinclude:: pe2.bib
     :language: bibtex
 
+.. literalinclude:: pe3.bib
+    :language: bibtex
+
 Please see https://github.com/fjxmlzn/private-evolution-papers for the full list of **Private Evolution** papers and code repositories done by the community.

doc/source/getting_started/pe1.bib

@@ -1,4 +1,4 @@
-@article{lin2023differentially,
+@article{pe1,
   title={Differentially private synthetic data via foundation model apis 1: Images},
   author={Lin, Zinan and Gopi, Sivakanth and Kulkarni, Janardhan and Nori, Harsha and Yekhanin, Sergey},
   journal={arXiv preprint arXiv:2305.15560},

doc/source/getting_started/pe2.bib

@@ -1,4 +1,4 @@
-@article{xie2024differentially,
+@article{pe2,
   title={Differentially private synthetic data via foundation model apis 2: Text},
   author={Xie, Chulin and Lin, Zinan and Backurs, Arturs and Gopi, Sivakanth and Yu, Da and Inan, Huseyin A and Nori, Harsha and Jiang, Haotian and Zhang, Huishuai and Lee, Yin Tat and others},
   journal={arXiv preprint arXiv:2403.01749},

doc/source/getting_started/pe3.bib

@@ -0,0 +1,6 @@
+@article{pe3,
+  title={Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Models},
+  author={Lin, Zinan and Baltrusaitis, Tadas and Yekhanin, Sergey},
+  journal={arXiv preprint arXiv:2502.05505},
+  year={2025}
+}