ASCII fast reader weird behaviour for comments

[astronomyk]

Hello again!

As always, thanks a ton for all your hard work with astropy! Couldn't live without it =)

Just thought I'd give you another quick heads up to some weird border-case behaviour for the ASCII fast-reader. (Related to issue #8267).

It seems that when I include an additional comment line below my column headers (to keep track of units), this breaks the comment parser of the fast-reader.

This works:

In [159]: ascii.read("""
     ...: # key1 : val 1 
     ...: # key2 : extra long entry
     ...: # key3 : also super long entry
     ...: col1    col2
     ...: 0       1
     ...: """).meta
Out[159]: 
OrderedDict([('comments',
          ['key1 : val 1',
           'key2 : extra long entry',
           'key3 : also super long entry'])])

This doesn't:

In [160]: ascii.read("""
     ...: # key1 : val 1 
     ...: # key2 : extra long entry
     ...: # key3 : also super long entry
     ...: col1    col2
     ...: # unit1 unit2
     ...: 0       1
     ...: """).meta
Out[160]: 
OrderedDict([('comments',
              ['key1 : val 1',
               'key2 : extra long entry',
               'key3 : al',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               'unit1 unit2'])])

Additionally when I shortened the length of line 2 (--> # key2 : short), the amount of text removed for line 3 (and subsequent lines) changes in proportion to the amount of characters in line 2.

In [164]: ascii.read("""
     ...: # key1 : val 1 
     ...: # key2 : short
     ...: # key3 : also super long entry
     ...: col1    col2
     ...: # unit1 unit2
     ...: 0       1
     ...: """).meta
Out[164]: 
OrderedDict([('comments',
              ['key1 : val 1',
               'key2 : short',
               'key3 : also super lo',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               '',
               'unit1 unit2'])])

No idea what's going on here though. As stated, my work-around is just to set fast_reader = False, then everything works. Just thought I'd bring it to your attention.

Cheers!

[pllim]

Thanks for the report and kind words, @astronomyk ! cc @taldcroft

[pllim]

Also, what Astropy and Python versions are you using?

[astronomyk]

I reformatted my operating system on Monday, so everything is brand new from pip

• Python 3.6.8
• Astropy 3.1.1