Setting masked pixel to NaN

[jo-moeller]

Would it be possible to change the value to which masked pixels are generally set to nan instead of zero? I see the following possible improvements:

• Mean, median (maybe also var and std in the future) are not correct now, as masked pixel should not be taken into account for such calculations. It would work correctly however with nanmean and nanmedian.
• According to Jola, it will be optional in the future that the correction pipeline already masks bad pixel. These pixels would/could be set to nan and extra foam could already handle these masked pixels.
• Projections would not show artifacts from detector gaps.
• Histograms would work correctly and not have an artificial peak at 0.

Especially the first and the last points are mandatory, if we want to do online speckle analysis with extra foam in the future.

Cheers,
Johannes

[zhujun98]

Thanks for the feedback! Indeed, this is an important feature in 0.8.0 and I am finalizing the implementation in

#61

Mean, median and std will be calculated using np.nanmean, np.nanmedian, np.nanstd in the coming release.

I used to think that pixels with negative values are not useful and should be masked as Nan, then one can bypass the masking issue by specifying a positive number for the lower boundary of histogram.

Concerning the 3rd point, do you mean that I should use np.nansum(data, axis=-1) instead of np.sum(data, axis=-1) to calculate the projection?

To be frankly, I have been obsessed by the Nan issue for a long time.

[jo-moeller]

Ah, nice. I missed it because it's already closed.

Concerning the histograms for AGIPD, this would however not work by just setting the pixels to a negative number. The noise peak of AGIPD has a width of ~10 and is centered around 0. So having values like -5 is absolutely valid and should not be discarded. One quantity that we look at is for example the height of the peak around 0 compared to the height of the peak around 65. So the range would be for example -100 to 250 of the histogram. Is it the normal numpy histogram function? That also works with NaNs.

For the projection, yes. I would vote for using nansum there as well. (Maybe an option where we can also use nanmean, as we are on the topic :) )

[zhujun98]

Thanks for the explanation!

Is it the normal numpy histogram function? That also works with NaNs.

Yes. For now, we use np.histogram to calculate the histogram.

Maybe an option where we can also use nanmean, as we are on the topic.

Sure. I opened an issue #68. Most probably it will go into 0.8.0. Otherwise, one week later in 0.8.1.