Label-Free Model Evaluation with Semi-Structured Dataset Representations

Prerequisites

This code uses the following libraries

• Python 3.7
• NumPy
• PyTorch 1.7.0 + torchivision 0.8.1
• Sklearn
• Scipy 1.2.1

---

Data Preparation

Thanks to Deng Weijian for providing the code for generating sample sets. Please refer to https://github.com/Simon4Yan/Meta-set, to generated datasets to train regression model. The newly collected datasets are avalibale at link1 and link2.

---

Run the Code

1. Creat sample sets and 2. Train classifier and get image features of sample sets

   pleaser refer to

   https://github.com/Simon4Yan/Meta-set/blob/main/meta_set

2. Get set representations

   # get shape, clusters and sampled data.  
   python Set_rep/get_set_representation.py

3. Train the regresssor on dataset representaions

   python Set_rep/train_regnet_new.py

4. Test the regresssor

   python Set_rep/test_regnet_new.py

---

Citation

If you use the code in your research, please cite:

@article{DBLP:journals/corr/abs-2108-10310,
  author    = {Xiaoxiao Sun and
               Yunzhong Hou and
               Hongdong Li and
               Liang Zheng},
  title     = {Label-Free Model Evaluation with Semi-Structured Dataset Representations },
  journal   = {CoRR},
  volume    = {abs/2108.10310},
    url       = {https://arxiv.org/abs/2108.10310}
  year      = {2021},
}

@inproceedings{deng2020labels,
author={Deng, Weijian and Zheng, Liang},
title     = {Are Labels Always Necessary for Classifier Accuracy Evaluation?},
booktitle = {Proc. CVPR},
year      = {2021},
}