Evidential Uncertainty Calibration

[muammar]

Very nice paper. I've been facing similar issues as the ones reported in this figure when using evidential uncertainty quantification:

In your work, you discuss we could add an Accuracy vs. Uncertainty loss function that looks like this:

You proposed a new version of it, as shown in the equation below:

I could find the implementation of the equation above here:

DEAR/mmaction/models/losses/edl_loss.py

Lines 113 to 144 in 2a64f6a

	def edl_loss(self, func, y, alpha, annealing_coef, target):
	"""Used for both loss_type == 'log' and loss_type == 'digamma'
	func: function handler (torch.log, or torch.digamma)
	y: the one-hot labels (batchsize, num_classes)
	alpha: the predictions (batchsize, num_classes)
	epoch_num: the current training epoch
	"""
	losses = {}
	S = torch.sum(alpha, dim=1, keepdim=True)
	A = torch.sum(y * (func(S) - func(alpha)), dim=1, keepdim=True)
	losses.update({'loss_cls': A})
	losses.update({'lambda': annealing_coef})
	if self.with_kldiv:
	kl_alpha = (alpha - 1) * (1 - y) + 1
	kl_div = annealing_coef * \
	self.kl_divergence(kl_alpha)
	losses.update({'loss_kl': kl_div})
	if self.with_avuloss:
	pred_scores, pred_cls = torch.max(alpha / S, 1, keepdim=True)
	uncertainty = self.num_classes / S
	acc_match = torch.reshape(torch.eq(pred_cls, target.unsqueeze(1)).float(), (-1, 1))
	if self.disentangle:
	acc_uncertain = - torch.log(pred_scores * (1 - uncertainty) + self.eps)
	inacc_certain = - torch.log((1 - pred_scores) * uncertainty + self.eps)
	else:
	acc_uncertain = - pred_scores * torch.log(1 - uncertainty + self.eps)
	inacc_certain = - (1 - pred_scores) * torch.log(uncertainty + self.eps)
	avu_loss = annealing_coef * acc_match * acc_uncertain + (1 - annealing_coef) * (1 - acc_match) * inacc_certain
	losses.update({'loss_avu': avu_loss})
	return losses

It is unclear to me when to use the disentangle case. Could you please provide me with any insights about when to use one over the other?

Thanks :)

[Cogito2012]

Good question!
Actually their purposes are the same. For the disentangle case, the two terms logp(1-u) and log(1-p)u will reduce to four individual terms logp, log(1-u), log(1-p), and logu. However, in this case, the variables p and u are still correlated in SGD optimization because p and u are derived from the same DNN output alpha. I did not choose the disentangle case by default simply because it does not work better than the other one in experiments. Intuitively, the Eq. (3) in the paper is more like a cross-entropy between p and u, which is a more common practice.

You may also refer to this NeurIPS 2020 paper to see how it deal with the AvU calibration:
"Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. In
NeurIPS, 2020."

[muammar]

Thanks for your fast response, I appreciate it. I will use the version with disentangle set to False. Thanks for sharing the reference (I had started reading it).