filtering masked tables resets column fill_value

[sbailey]

filtering a masked table resets the fill value to 1:

In [1]: from astropy.table import Table

In [2]: a = Table(masked=True)

In [3]: a['X'] = [0,1,2,3]

In [4]: a['Y'] = [0,1,2,3]

In [5]: a['Y'].mask = [False, False, False, True]

In [6]: print a['Y'].fill_value
999999

In [7]: a
Out[7]: 
<Table masked=True length=4>
  X     Y  
int64 int64
----- -----
    0     0
    1     1
    2     2
    3    --

In [8]: a = a[a['Y']<3]

In [9]: print a['Y'].fill_value
1

This is dangerous because it reset the fill value to a valid value in the table, which breaks how masked values are tracked in FITS-format tables (I'll submit a separate issue about that). But filtering shouldn't reset the fill value in the first place.

In [10]: a
Out[10]: 
<Table masked=True length=3>
  X     Y  
int64 int64
----- -----
    0     0
    1     1
    2     2

In [11]: a.write('blat.fits', overwrite=True)

In [12]: b = Table.read('blat.fits')

In [13]: b
Out[13]: 
<Table masked=True length=3>
  X     Y  
int64 int64
----- -----
    0     0
    1    --
    2     2

The end result is that after filtering my masked table, round-tripping it to a FITS file gives me back a different table, with one of my previously unmasked values now masked.

[pllim]

Maybe less confusing to just address this in #4708 since they are related?

[sbailey]

Fine to treat this as a single ticket if both #4707 and #4708 are fixed simultaneously. But they are separate issues — resetting the fill value when filtering could be problematic even if no I/O is ever performed with the table. It may be possible to usefully fix this ticket by itself even if the FITS I/O issue #4708 requires more work.

[pllim]

Ok. Re-opening. Thanks for the clarifications.

[saimn]

Seems to be fixed, maybe it was a Numpy issue with subclasses ?

[astropy-bot]

Hi humans 👋 - this issue was labeled as Close? approximately 5 months ago. So..... any news? 🗞️

If you think this issue should not be closed, a maintainer should remove the Close? label - otherwise, I'm just gonna have to close this issue in a month. Your time starts now! Tick tock 🕙

If you believe I commented on this issue incorrectly, please report this here

[saimn]

Ok, closing as I still cannot reproduce.

[sbailey]

I confirm that this problem still exists with astropy/2.0.2 + numpy 1.11.1 but it is fixed with astropy/2.0.2 + numpy/1.13.1 . OK with me to leave it closed.