saliency for reconstruction task

[james20141606]

Thanks for the great implementation! I am wondering what would be different if I'd like to investigate the saliency map of a reconstruction task. In the classification task you can select one neuron, but for reconstruction task, the output is the whole image, is it possible to visualize it in similar way?

[tolga-b]

Hi Chen,
Thanks for reaching out. Do you mean tasks such as image infilling or super-resolution when you say reconstruction? I haven't looked into how saliency looks like for reconstruction models, but one idea is seeing each pixel as a separate class which would give you a heatmap of where the model is looking at when predicting that particular pixel. You can maybe sample a subset of pixels in the image on a grid and visualize heatmaps at each location separately.

[james20141606]

Hi Chen,
Thanks for reaching out. Do you mean tasks such as image infilling or super-resolution when you say reconstruction? I haven't looked into how saliency looks like for reconstruction models, but one idea is seeing each pixel as a separate class which would give you a heatmap of where the model is looking at when predicting that particular pixel. You can maybe sample a subset of pixels in the image on a grid and visualize heatmaps at each location separately.

Thanks for your inspiring reply! for reconstruction I mean some auto-encoder/GAN based image to image reconstruction. Your idea to use each pixel as class is worth trying, but might still have some problem: we may treat them as continuos value instead of discrete value. So the problem became: could we change the code for regression task visualization?

[tolga-b]

I think it should work out of the box as long as you pass the regression neuron as the output neuron for the saliency. Please let me know if you run into any issues!