back-propagate the discriminator loss through the generator

[mehdimashayekhi]

Hi , Thanks for sharing your code. I have a quick question, in the paper it is mentioned that We don’t back-propagate the discriminator loss through the generator, maybe I have missed it, but where in your code this has been taken care of ? can you refer me to it ? Thanks

[richarddwang]

Hi.
There is a sampling between generator and discriminator and sampling is not able to be backward propagated.

electra_pytorch/pretrain.py

Lines 294 to 302 in 9b2533e

	with torch.no_grad():
	# sampling
	pred_toks = self.sample(mlm_gen_logits) # ( #mlm_positions, )
	# produce inputs for discriminator
	generated = masked_inputs.clone() # (B,L)
	generated[is_mlm_applied] = pred_toks # (B,L)
	# produce labels for discriminator
	is_replaced = is_mlm_applied.clone() # (B,L)
	is_replaced[is_mlm_applied] = (pred_toks != labels[is_mlm_applied]) # (B,L)