Exploiting Connections between Lipschitz Structures for Certifiably Robust Deep Equilibrium Models

Paper | Bibtex

Abstract

Recently, deep equilibrium models (DEQs) have drawn increasing attention from the machine learning community. However, DEQs are much less understood in terms of certified robustness than their explicit network counterparts. In this paper, we advance the understanding of certified robustness of DEQs via exploiting the connections between various Lipschitz network parameterizations for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures, including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special cases of the Lipschitz-bounded equilibrium networks (LBEN) without changing the prescribed Lipschitz constant in the original network parameterization. A key feature of our reparameterization technique is that it preserves the Lipschitz prescription used in different structures. This opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-based fine-tuning. We also support our theoretical understanding with new empirical results, which show that our proposed method improves the certified robust accuracy of DEQs on classification tasks.

Code

Train & evaluation of the SLL model

mkdir -p ./trained_models/sll_models

# training
torchrun --standalone --nnodes=1 --nproc_per_node=gpu main.py --train_dir ./trained_models/sll_models --data_dir ./data --mode train-sll --lr 0.01 --epochs 1000 

# evaluation
torchrun --standalone --nnodes=1 --nproc_per_node=gpu main.py --mode eval-sll --data_dir ./data --train_dir ./trained_models/sll_models

Finetune DEQ from the SLL model

mkdir -p ./trained_models/deq_finetune

# finetunning of the SLL model from DEQ architecture
torchrun --standalone --nnodes=1 --nproc_per_node=gpu main.py --train_dir ./trained_models/deq_finetune --data_dir ./data --mode finetune-deq \
	--gradient_clip_by_norm 1 --ckpt_path ./trained_models/sll_models/checkpoints/model.ckpt-196000.pth --lr 0.0000001 --kernel_reg 0.1 --kernel_reg_norm 1

# evaluation
torchrun --standalone --nnodes=1 --nproc_per_node=gpu main.py --mode eval-deq --data_dir ./data --train_dir ./trained_models/deq_finetune --ckpt_path ./trained_models/sll_models/checkpoints/model.ckpt-196000.pth

Bibtex

@inproceedings{havens2023exploiting,
  title={Exploiting Connections between Lipschitz Structures for Certifiably Robust Deep Equilibrium Models},
  author={Havens, Aaron J and Araujo, Alexandre and Garg, Siddharth and Khorrami, Farshad and Hu, Bin},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
  year={2023}
}