inference code

[wdeback]

Dear Yongchan,

I've been reading your paper and code with great interest. Seems like a very interesting way to assess predictive uncertainty in DNN models for segmentation, based purely on inference-time Dropout. However, I can't seem to find the code in which you do inference and actually compute the aleatoric and epistemic uncertainties.

Specifically, the implementation of the key eq. 4 in the paper seems to be missing. Could you provide this or point me to it?

Thanks for this interesting work!

[ykwon0407]

First of all, thank you for your interest!

At the line 18 in the model.py, the class Dropout_uncertain forces to randomly drop nodes at inference time. So, at the line 152~156 in train.py, model.predict() will compute random predictions, that is in the paper.

As you pointed out, the current version does not include the code for computing aleatoric and epistemic uncertainties. (I am going to upload the updated version around August!!) If you have a set of probability estimates , a numpy array p_hat with dimension (N_uncertain, width, height, depth), then the epistemic and aleatoric uncertainties can be obtained by the following code.

epistemic = np.mean(p_hat**2, axis=0) - np.mean(p_hat, axis=0)**2
aleatoric = np.mean(p_hat*(1-p_hat), axis=0)

Thank you!!

[wdeback]

Thanks for your helpful reply.

With your help, I was able to reproduce your results on a retinal blood vessel segmentation problem based on the DRIVE dataset.

You can check out the notebook here: https://gitlab.com/wdeback/dl-keras-tutorial/blob/master/notebooks/3-cnn-segment-retina-uncertainty.ipynb. I can send some results, if you're interested.

One minor thing: there is no need anymore to define test-time Dropout as a custom layer as you do here, since it is built-in in keras: keras-team/keras#9412 (comment).

Thanks for sharing!

[ykwon0407]

Sure, I'd love to see it! Could I get via email? my email address is 'ykwon0407[at]snu[dot]ac[dot]kr'.
And thank you for your comment. It is really helpful!

[fatemehtd]

Dear Yongchan,
Could you please let me know whether the eq. 4 in the paper is applicable for multi label segmentation or just the binary segmentation?

[ykwon0407]

Dear redsadaf,

Thank you for your interests! The eq.4 in the paper is defined for multi-label segmentation. So you can apply the equation for not only binary segmentation but multi-class segmentation problems. Please note that if the eq.4 will provide a K by K matrix if there are K categories in your dataset.
.
In the case of binary classification (when K=2), the eq.4 produces a 2 by 2 matrix. However, the two diagonal elements are just the same each other, and similarly, two different off-diagonal terms are also same. Thus, we obtain numeric values, not matrices, for uncertainty maps.
.
Please let me know if you have any further questions and hope this is informative!!

[mongoose54]

@ykwon0407 Could you elaborate how the formulas:
epistemic = np.mean(p_hat**2, axis=0) - np.mean(p_hat, axis=0)**2 ,and
aleatoric = np.mean(p_hat*(1-p_hat), axis=0)
came up from equation 4?
I cannot seem to wrap my head around the aleatoric term: aleatoric = np.mean(p_hat*(1-p_hat), axis=0)

[ykwon0407]

@mongoose54
Hello~~ Please note that the source code is for the binary segmentation problem!! That is, the two diagonal terms in the 2 by 2 aleatoric matrix have the same value each other, say p*(1-p).
Thus, I first calculated p_hat* (1-p_hat) and average it across the random samples. Hope this is informative!

[tamkaho]

In the case of multi-class segmentation, how would you interpret the diagonal / off diagonal elements of the matrix in Eq. 4? Is it just like co-variance between the different class predictions?

[ykwon0407]

@tamkaho Yes, it is. Thank you for your interest!