This repository contains all the code used in the following paper:

• Kanrar, R., Jiang, F., Cai, Z. (2025). Model-free Change-point Detection Using AUC of a Classifier. Journal of Machine Learning Research Journal/arXiv

Contents

• /code: This folder contains all the R and Python code used in the paper.
• /output: This folder contains the output saved in various steps.
• /data: This folder contains data used in the paper. Data sets are already stored, and they can be processed with reproducible code available in the folder.
• .gitattributes: This file is auto-generated by Git LFS to track large files.
• .gitignore: This file is auto-generated by Git to track unnecessary files not added to the repository.
• changeAUC_exec.sh: This is an executable file to run all the simulation experiments for Euclidean data in Section 4.1 using the changeAUC method.
• gSeg_exec.sh: This is an executable file to run all the simulation experiments for Euclidean data in Section 4.1 using the gSeg method.

The following paragraphs outline an overall description of the main sub-folders in this repository.

Contents of /code/r/:

• /generate_data: contains R scripts to simulate high dimensional Euclidean data as described in Section 4.1
• /get_change_point: contains R scripts for the main functions to detect single and multiple change points.
• /get_null_quantiles: generate and combine null quantiles for the proposed test statistic. See Table 1 for more details.
• /misc: contains some additional functions, such as calculation of Adjusted Rand Index (ARI), etc.
• /other_methods: contains R implementation of the other methods compared.
• /real_data: contains R scripts to access and process the real data sets used in Section 5.
• /simulation_scripts: contains R scripts used to conduct simulation experiments and Section 4 and to produce visualizations and tables.
• requirements.R: contains R packages required to install first.

Contents of /code/py/:

• /generate_data: contains Python scripts to simulate high dimensional Euclidean data as described in Section 4.1
• /get_change_point: contains Python scripts for the main functions to detect single and multiple change points.
• /misc: contains some additional functions, such as calculation of Adjusted Rand Index (ARI), generating heat maps for the NYC Taxi Data, etc.
• /real_data: contains Python scripts to access and process the real data sets used in Section 5.
• /simulation_scripts: contains Python scripts used to conduct simulation experiments and Section 4 and to produce visualizations and tables.

Contents of /data:

• This folder contains processed data sets used in the paper. We also provide reproducible code to access and process the data sets in the /real_data folder inside either /code/r/ or /code/py/.

Contents of /output:

• This folder contains output files created in the real data analysis. Other files created in the simulation experiments are omitted due to the size limit in the Github repository.

Initial Setups:

• Clone Github Repository:

git clone git@github.com:rohitkanrar/changeAUC.git
cd changeAUC

• Install all required R packages:

source("code/r/requirements.R")

• Create Python Environment:

conda create -n "hd_cpd" python=3.7.13
conda activate hd_cpd
pip install -r code/py/requirements.txt

Test Cases:

In the following, we include three test cases to check the validity of empirical results reported in Section 4 of the paper.

• The following command detects a change point using Random Forest with data generated by the Dense Mean setup with sample size 1000 and dimension 500 over 10 replications. The output will be created and saved in the /output/dense_mean/ folder. We set the seed to 2024 for illustrations only.

Approximate Time (CPU): 5 seconds.

Rscript code/r/simulation_scripts/simulation_changeAUC.R -d 2 -n 1000 -p 500 -r 10 -g "dense_mean" -l "local" -c "RF" -s 2024

• The following command performs the same task above but uses a Fully connected neural network (FNN).

Approximate Time (CPU): 30 seconds.

python code/py/simulation_scripts/simulation_changeAUC.py -d 2 -n 1000 -p 500 -r 10 -g dense_mean -l local -c FNN -s 2024

• The following command detects a change point using the vgg16 classifier with data simulated from the CIFAR10 database. The data includes 500 images of dogs followed by 500 images of cats. We replicate the procedure over 10 replications with seed 2024. The output will be created and saved in the /output/cifar/3-5/ folder. Here, 3 and 5 refer to the dog and cat, respectively, as arranged in the CIFAR10 database.

Approximate Time (CPU): 250 seconds.

python code/py/simulation_scripts/simulation_cifar.py -r 10 -n 1000 -g 35 -c VGG16 -l local -s 2024

Real Data Examples:

To run the real data examples presented in Section 5, we refer to following the steps outlined below.

US Stock Data

The following R script performs multiple change point detection using Seeded Binary Segmentation with changeAUC as the test statistic and two classifiers, Logis and Rf.

source("code/r/real_data/us_stocks/implement_script/changeAUC.R")

The following Python script performs the same task above with the Fnn classifier.

python code/py/real_data/us_stocks/implement_script/changeAUC.py

Implementation of the other competitive methods can be found in the respective folders of the above scripts.

NYC Taxi Data

The following Python code performs multiple change point detection using Seeded Binary Segmentation with changeAUC as the test statistic and vgg16 as the classifier.

python code/py/real_data/nyc_taxi/implement_script/changeAUC.py   

Acknowledgements

We thank the authors of Chakraborty, Zhang 2021+ for sharing the R code implementation of their proposed method. We also thank the Information Technology Services (ITS) at Iowa State for providing free access to the High Performance Computing (HPC) cluster `Pronto.'