Unable to read some NED VOTables with astropy.table.Table

[jdnc]

The errors specifically happen while retrieving VOTables for the NED References by Object Name queries. A sample VOTable that causes errors from this search is here.

For instance on doing this:

In [1]: import astropy.utils.data as aud
In [2]: from astropy.table import Table
In [3]: with aud.get_readable_fileobj('http://ned.ipac.caltech.edu/cgi-bin/datasearch?objname=m1&ref_extend=no&begin_year=1800&end_year=2013&of=xml_main&search_type=Reference') as f:
table = Table.read(f, format='votable')
Downloading http://ned.ipac.caltech.edu/cgi-bin/datasearch?objname=m1&ref_extend
Downloading http://ned.ipac.caltech.edu/cgi-bin/datasearch?objname=m1&ref_extend=no&begin_year=1800&end_year=2013&of=xml_main&search_type=Reference [Done]
WARNING: W42: None:2:0: W42: No XML namespace specified [astropy.io.votable.exceptions]
WARNING: W22: None:3:0: W22: The DEFINITIONS element is deprecated in VOTable 1.1.  Ignoring [astropy.io.votable.exceptions]
WARNING: W04: None:17:0: W04: content-type 'char' must be a valid MIME content type [astropy.io.votable.exceptions]
ERROR: ValueError: None:37:4: ValueError: invalid literal for int() with base 10: '1.' (in row 0, col 'ref_col1') [astropy.io.votable.exceptions]

The problem apparently seems to be caused by the fact that the entries in the column ref_col1 are of the form 1. rather than simply 1. So trying to parse 1. like entries throws the exception. Would there be any simple workaround to this?

[mdboom]

Strictly speaking, the file is violating the spec, so this should be reported to the maintainer of the tool that is producing the file. I've sent an e-mail to ned@ipac.caltech.edu about this and will keep this issue updated with the result.

That said, we do have a history of adding workarounds for this sort of thing to astropy.io.votable. But I'm loathe to do so unless the author of the data source is unable to be reached or unwilling to make the fix, as all of these workarounds degrade performance and complicate the code base and are ultimately a maintenance burden on us over time.

[embray]

Yeah, don't really want VOTable to become more like FITS than it needs to be 😝

[mdboom]

Just a heads up, still no reply from ned@ipac.caltech.edu.

[perrygreenfield]

I did ask, but only got pointed to the director of the place, which didn't seem that useful. I'll see if I can drill down a bit.

On Jul 31, 2013, at 2:41 PM, Michael Droettboom wrote:

Just a heads up, still no reply from ned@ipac.caltech.edu.

—
Reply to this email directly or view it on GitHub.

[perrygreenfield]

I dug up a contact and just sent an email.

[keflavich]

@perrygreenfield @mdboom - Any updates?

[mdboom]

I was told on the 01-08-2013 that the NED developers are looking into it.

[cdeil]

Has this issue been fixed in the meantime?

In [13]: with aud.get_readable_fileobj('http://ned.ipac.caltech.edu/cgi-bin/datasearch?objname=m1&ref_extend=no&begin_year=1800&end_year=2013&of=xml_main&search_type=Reference') as f: table = Table.read(f, format='votable')
Downloading http://ned.ipac.caltech.edu/cgi-bin/datasearch?objname=m1&ref_extend=no&begin_year=1800&end_year=2013&of=xmlDownloading http://ned.ipac.caltech.edu/cgi-bin/datasearch?objname=m1&ref_extend=no&begin_year=1800&end_year=2013&of=xml_main&search_type=Reference [Done]
WARNING: W42: None:2:0: W42: No XML namespace specified [astropy.io.votable.exceptions]
WARNING: W22: None:3:0: W22: The DEFINITIONS element is deprecated in VOTable 1.1.  Ignoring [astropy.io.votable.exceptions]

In [15]: table[0]
Out[15]: 
<Row 0 of table
 values=(1, '2012ApJS..201...33W', 'Wilson-Hodge, C...', 'Three Years of Fermi GBM Earth Occultation Monitoring: Observations of Hard X-Ra...')
 dtype=[('ref_col1', '<i4'), ('ref_col2', 'O'), ('ref_col3', 'O'), ('ref_col4', 'O')]>

[mdboom]

It looks like the NED developers have fixed this, I just never got notification of that.

I'm closing this bug as "fixed upstream".