Questions about the default setting of parameter "warmup_epoch"

[XiangQiu42]

Dear dayu11,

In your code, the parameter "warmup_epoch" is -1 by default, and it seems that you did not assign a new value to this parameter in your experiment.

However, if "warmup_epoch" is -1, it seems that at the train function:

Differentially-Private-Deep-Learning/vision/RGP/cifar_train.py

Lines 261 to 263 in 77096c8

	if(epoch + 1 == args.warmup_epoch):
	# take a snapshot of current model for computing historical update
	net.module.update_init_weight()

This part of code will never be executed.

So, Is that correct or I am missing something?

[dayu11]

Yes, you are correct. By default, the code directly decomposes the weight matrix itself instead of the historical update.

It turns out that directly decomposing the weight matrix also works. This may because as the training progresses, the weight matrix itself contains enough information about the historical update, i.e., we do not need to subtract the weight at initialization from the current weight to get the historical update. Still, you can set warmup_epoch>1 to compute the historical update explicitly.