Why is mask not applied to Spectrum1D when present?

[pllim]

I am basically re-asking the question in #142 that was closed without explanation. In spacetelescope/synphot_refactor#243 , I feel like it is not synphot's job to apply the mask for specutils. I expect specutils to mask out unwanted data when accessed if there is a mask. Why is this not the case? Thanks.

Take this as example:

>>> from astropy import units as u                                                             
>>> from specutils import Spectrum1D
>>> lamb = [0.001, 0.25, 0.5] * u.Hz                                                           
>>> flux = [1.5, 0.5, 99.9] * u.nJy
>>> mask = [True, False, False]                                                            
>>> spec = Spectrum1D(spectral_axis=lamb, flux=flux, mask=mask)                                           
>>> spec
 <Spectrum1D(flux=<Quantity [ 1.5,  0.5, 99.9] nJy>, spectral_axis=<Quantity [0.001, 0.25 , 0.5  ] Hz>)>

What I see:

>>> spec.flux                                                                                  
<Quantity [ 1.5,  0.5, 99.9] nJy>

What I thought I would see:

>>> spec.flux
<Quantity [ nan,  0.5, 99.9] nJy>

While I understand that Astropy does not yet support masked quantity natively (xref astropy/astropy#1852), perhaps specutils can work around that limitation for now?

[eteq]

I'm not clear why #142 was closed, but hopefully this answer will suffice:

The mask attribute in Spectrum1D comes from the mask in nddata - you can see the full documentation here: https://docs.astropy.org/en/stable/api/astropy.nddata.NDData.html. Its purpose is to store contextual information about whether a given pixel is valid, even if the data in that pixel still has some meaning. For example, a hot pixel is still useful if the flux in it is much higher than how "hot" it is, so depending on the exact context the value in it may or may not be useful.

So it's a close analogy with numpy masked arrays. The mask tells you "is this pixel masked", while the .flux (which is the same as .data in numpy masked arrays) is the value of the pixel. By contrast, the nan-masking approach in the "What I thought I would see" completely overwrites the underlying value, instead of giving an option to access it if need be.

So the idiom intended in specutils (just like NDData) is that the onus is on the user to interpret the mask. In #516 there's some discussion of how this is supposed to work in specutils functions, but the gist is generally "use [mask==False] if you want only the valid points".

Does that answer the question? If so I think this issue should be left open until something about the above can be put in the specutils docs, since this should definitely be in there!

[pllim]

I guess I disagree that it should fall to the user to interpret the mask in specutils. As a user, if I give you a mask, I expect my result to be automatically masked. That said, I have also not used NDData much directly. I'll defer to opinions of other specutils stakeholders.

[eteq]

@pllim - a middle option discussed else where was to implement a flux_masked attribute which would behave the way you suggest. But there's some discussion here: #509 (comment) that explains how that turns out to be problematic - NaNs are "infectious", and tend to break a lot of analysis operations if you try to use them blindly. So in the use case described there (and a couple others that have come later) the idea is "always carry around a Spectrum1D which knows its own mask".

The trouble in spacetelescope/synphot_refactor#243 is that a Spectrum1D is being mapped onto a different kind of spectrum (snyphot's SourceSpectrum), which doesn't have a formal concept of a mask at all. So it's sort of an impedance mis-match between those.

There is yet another alternative we have not yet discussed which could be done: have a method or attribute or something along the lines of Spectrum1d.get_masked_flux_and_spectral_axis() which does the masking and yields the appropriate flux and spectral axes (which is what spacetelescope/synphot_refactor#243 needs). But it's not clear to me if that's really any clearer than spec.flux[spec.mask==False], spec.spectral_axis[spec.mask==False] would be?