FITS BINTABLE HDUs needless strip trailing whitespace from string fields

[embray]

Description

This is a follow-up to #11312.

Expected behavior

When saving string data in a FITS binary table column, the data should be preserved as-written (assuming the data fits in the column width).

Actual behavior

However, if a string contains trailing whitespace (particularly spaces, as FITS allows any printable ASCII characters in text columns) that trailing whitespace is removed, even going so far as to modify the data of existing FITS files when opened in update mode (#11312).

Steps to Reproduce

The following demonstrates the problem simply:

>>> from astropy.io import fits
>>> import numpy as np
>>> data = np.array([b'abc', b'ab ', b'a  ', b'   '], dtype=[('a', 'S3')])
>>> data['a']
array([b'abc', b'ab ', b'a  ', b'   '], dtype='|S3')
>>> hdu = fits.BinTableHDU.from_columns(data)
>>> hdu.header
XTENSION= 'BINTABLE'           / binary table extension                         
BITPIX  =                    8 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                    3 / length of dimension 1                          
NAXIS2  =                    4 / length of dimension 2                          
PCOUNT  =                    0 / number of group parameters                     
GCOUNT  =                    1 / number of groups                               
TFIELDS =                    1 / number of table fields                         
TTYPE1  = 'a       '                                                            
TFORM1  = '3A      '                                                            
>>> hdu.writeto('a.fits')
>>> with fits.open('a.fits') as hdul:
...     print(hdul[1].data)
...     print(hdul[1].data.tobytes())
... 
[('abc',) ('ab',) ('a',) ('',)]
b'abcab\x00a\x00\x00 \x00\x00'

The last line demonstrates that this is not just an issue of how the string field is displayed when printing: the underlying bytes are modified as well. Weirdly, for the last row, a single leading is kept but the following ones are replaced with zeros.

Additional Background

The current behavior is as designed. Previously there was a belief that this might be required by the FITS standard, but actually we can find no evidence that that is the case in the latest draft (if it was in a previous draft I can't find that either). However, the existing functionality has been in place--probably just a side effect of implementation details of Numpy at the time--since early versions of PyFITS.

Since the existing functionality has been in place for such a long time, care needs to be taken in removing it.

Proposed Solution

See #11312 (comment)