Probability correction function does not seem to be correct

[hari-kk]

In the probability correction algorithm, the generated labels should be multinoimally distributed among all the aviable labels. However according to the code given, it will just be 0 or 1

The issue come because the code

PLC/utils.py

Lines 336 to 341 in 8c2bcfd

	for i in range(f_x.shape[0]):
	cur_prob_distri = f_x[i]
	cur_prob_distri = softmax(cur_prob_distri)
	top_k_idx = np.argsort(cur_prob_distri)[-1:]
	top_probs = cur_prob_distri[top_k_idx]

will result in a scalar top_probs. Hence the code to generate new labels

PLC/utils.py

Lines 349 to 353 in 8c2bcfd

	else:
	top_probs = top_probs / np.sum(top_probs) # normalization
	flipped = flipper.multinomial(1, top_probs, 1)[0]
	new_label = np.where(flipped == 1)[0]
	new_label = top_k_idx[new_label[0]] # new_label shape [1, ]

will result in new_label being just 0 or 1.

Moreover the normalization here

PLC/utils.py

Line 350 in 8c2bcfd

top_probs = top_probs / np.sum(top_probs) # normalization

will result in top_probs always being 1.0 thereby invalidating the whole idea of sampling from distribution.

Is this intended behavior?

[pingqingsheng]

"lrt_correction" and "prob_correction" are actually the same. We didn't clean our code and due to some historical reason there are two versions of such implementations, which is what we present in our paper. In other words, in the current version both functions achieve the same thing. We will clean it and update the code later.

We once wanted to try "prob_correction" which stochastically flip labels according to confidence, but it doesn't help a lot. You can try that yourself by changing the line "top_k_idx = np.argsort(cur_prob_distri)[-1:] " to "top_k_idx = np.argsort(cur_prob_distri)".