Crash on read_html(url, flavor="bs4") if table has only one column

[boarpig]

I was trying to read a package tracking table from finnish post office's website and I got

Traceback (most recent call last):
  File "./posti.py", line 69, in <module>
    dfs = read_html(html)
  File "/usr/lib/python3.4/site-packages/pandas/io/html.py", line 851, in read_html
    parse_dates, tupleize_cols, thousands, attrs, encoding)
  File "/usr/lib/python3.4/site-packages/pandas/io/html.py", line 721, in _parse
    infer_types, parse_dates, tupleize_cols, thousands))
  File "/usr/lib/python3.4/site-packages/pandas/io/html.py", line 609, in _data_to_frame
    _expand_elements(body)
  File "/usr/lib/python3.4/site-packages/pandas/io/html.py", line 586, in _expand_elements
    lens = Series(lmap(len, body))
  File "/usr/lib/python3.4/site-packages/pandas/compat/__init__.py", line 87, in lmap
    return list(map(*args, **kwargs))
TypeError: len() of unsized object

I isolated the offending table into this script:

https://gist.github.com/boarpig/de4044f4188fac700c68

The problem seems to be related to parse_raw_thead function

def _parse_raw_thead(self, table):
    thead = self._parse_thead(table)
    res = []
    if thead:
        res = lmap(self._text_getter, self._parse_th(thead[0]))
    return np.array(res).squeeze() if res and len(res) == 1 else res

Where res contains ['Tapahtumat'] which comes out of numpy array creation as

array('Tapahtumat', dtype='<U10')

which then produces previously mentioned error because you cannot take a len from that.

[boarpig]

I've been investingating this problem and I narrowed the problem down. Basically if you use BeautifulSoup4 as the backend and you have a table with table header with only one column, the _parse_raw_thead will cause the aforementioned error. In my case it was happening because there was multiple of same ids in the table body which caused lxml to error and switch to bs4.

I wonder why the whole np.array(res) even exists. Below is the simplest valid table that will cause the aforementioned error when doing pandas.read_html(html, flavor=bs4)

 <table>
    <thead>
        <tr>
            <th>Header</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>first</td>
        </tr>
    </tbody>
</table>

[jreback]

looks like a bug. care to do a pull-request?

[boarpig]

I would do but I'm not familiar enough with the code to fix this. The author obviously had a reason to do

return np.array(res).squeeze() if res and len(res) == 1 else res

instead of simply returning return res. Perhaps the assumption was that if len(res) is 1 it must be nested list like [['first', 'second']]and you want ['first', 'second']. Perhaps someone with more insingh can help with this. Using np.array().squeeze() seems like an odd way to flatten a list.

[cpcloud]

this could be fixed by adding np.atleast_1d after the squeeze call

[boarpig]

Here's a pull request to fix this
#9194