>AttributeError: 'NoneType' object has no attribute 'find'< on certain connections

[PattaFeuFeu]

I tried using Schiene as a home assistant sensor but after the first try—it did work for a brief amount of time—, the underlying library stopped working.

The problematic connection for me has been from Leverkusen Mitte to Köln Hansaring

Home Assistant Log

deutsche_bahn: Error on device update!
Traceback (most recent call last):
  File "/usr/src/app/homeassistant/helpers/entity_component.py", line 397, in _async_add_entity
    yield from entity.async_device_update(warning=False)
  File "/usr/src/app/homeassistant/helpers/entity.py", line 308, in async_device_update
    yield from self.hass.async_add_job(self.update)
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 327, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 250, in _wakeup
    future.result()
  File "/usr/local/lib/python3.6/asyncio/futures.py", line 243, in result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/app/homeassistant/components/sensor/deutsche_bahn.py", line 78, in update
    self.data.update()
  File "/usr/src/app/homeassistant/components/sensor/deutsche_bahn.py", line 99, in update
    self.start, self.goal, dt_util.as_local(dt_util.utcnow()))
  File "/usr/local/lib/python3.6/site-packages/schiene/schiene.py", line 135, in connections
    return parse_connections(rsp.text)
  File "/usr/local/lib/python3.6/site-packages/schiene/schiene.py", line 35, in parse_connections
    data = parse_delay(data)
  File "/usr/local/lib/python3.6/site-packages/schiene/schiene.py", line 57, in parse_delay
    delay_departure_raw = soup.find('div', class_="routeStart").find('span', class_="delay")
AttributeError: 'NoneType' object has no attribute 'find'

Command Line

I tried using Schiene directly using the command line but the same error arose.

>>> import schiene
>>> s = schiene.Schiene()
>>> s.connections('Leverkusen Mitte', 'Hansaring, Köln')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/schiene/schiene.py", line 135, in connections
    return parse_connections(rsp.text)
  File "/usr/local/lib/python3.6/site-packages/schiene/schiene.py", line 35, in parse_connections
    data = parse_delay(data)
  File "/usr/local/lib/python3.6/site-packages/schiene/schiene.py", line 57, in parse_delay
    delay_departure_raw = soup.find('div', class_="routeStart").find('span', class_="delay")
AttributeError: 'NoneType' object has no attribute 'find'

[kennell]

It looks like some reason a different HTML source code is returned when the search involves Hansaring, Köln as the destination station. When it is the origin station, it seems to work fine.

Edit: It looks like this bug only occurs in the latest release 0.21. The previous release 0.20 seems to give the correct result. So i guess the bug was introduced in this pull request here: #13. Unfortunately i currently do not have the time to dig deeper right now. Perhaps @akloeckner can take a look at this?

As a temporary workaround i suggest downgrading to 0.20.

[PattaFeuFeu]

Hm, according to https://github.com/home-assistant/home-assistant/blob/master/homeassistant/components/sensor/deutsche_bahn.py, version 0.20 is specified already?

[PattaFeuFeu]

… also, it’s currently working again.

[mkuhlmann]

I have the same problem for a connection inside of bonn and pinpointed the problem. It seems if you load the detail page for a connection, it opens collapsed for connections inside the same network/"verbund" (vrs/bonn for me).

Compare !details=opened! in the detail page url

to !details=opened!detailsVerbund=opened! in detail page url

This also fixed Leverkusen Mitte to Köln Hansaring (also vrs) as it is the same error, I just did a quick hack and did a search replace for the parameters in the url, will send a pull request.

[kennell]

Merged fix from @mkuhlmann and released new package on PyPI as version 0.22

[lud-hu]

Hey guys,

I've installed version 0.22 but the bug seem to persist for some connections.
Since I've just searched for a connection with delay I found this:

`

import schiene
s = schiene.Schiene()
s = s.connections('Augsburg','Augsburg-Oberhausen')
Traceback (most recent call last):
File "", line 1, in
File "/home/pi/.local/lib/python3.5/site-packages/schiene/schiene.py", line 135, in connections
return parse_connections(rsp.text)
File "/home/pi/.local/lib/python3.5/site-packages/schiene/schiene.py", line 35, in parse_connections
data = parse_delay(data)
File "/home/pi/.local/lib/python3.5/site-packages/schiene/schiene.py", line 65, in parse_delay
delay_arrival_raw = soup.find('div', class_="routeEnd").find('span', class_=["delay", "delayOnTime"])
AttributeError: 'NoneType' object has no attribute 'find'

`

Maybe the html helps you debugging:
bahn.html.zip

[derhuerst]

Hey!

A while ago I've written hafas-client, a JavaScript client for the HAFAS mgate.exe APIs used by e.g. the DB app. public-transport-enabler, the open source core behind the Öffi app also supports it.

It would be cool to build a Python port/clone, so that all of you can fetch HAFAS data from the environment you use. Let me know if you need assistance.

[dominik-th]

The last time I had parsing issues with this library I rewrote Schiene to use the HAFAS api (thanks for your hafas-client @derhuerst) but keep the same functions and their signatures so it works as a drop in replacement.

Maybe you have use for this: https://github.com/dominik-th/schiene2

Sample output of a connection with delay:
https://gist.githubusercontent.com/dominik-th/ad403c54fc0852952688eb0bc3ed9a05/raw/1b5db3cdc1ccd5e5befbd7fa0d07dac235842e6c

(I've never worked with python before, so please take a look on the code and fix things if you find anything)

[derhuerst]

The last time I had parsing issues with this library I rewrote Schiene to use the HAFAS api (thanks for your hafas-client @derhuerst) but keep the same functions and their signatures so it works as a drop in replacement.

I appreciate the effort, but arguably it's lacking quite a few features. If you want, we can make this a fully featured client.

[lud-hu]

Thanks to you both!

Nice to hear that there are some projects going on at the moment - and that they are not relying on parsing HTML. :D
I will try schiene2 and have a look on the JS-hafas-client. But I think schiene2 will already serve my demands on just displaying the next few departures with delays (and I need it in python).

[derhuerst]

I will try schiene2 and have a look on the JS-hafas-client. But I think schiene2 will already serve my demands on just displaying the next few departures with delays (and I need it in python).

Keep in mind that schiene2 doesn't (seem to) handle all edge cases, even when just fetching departures.

[lud-hu]

Yeah I've just seen that while playing around a little. One connection that was canceled was not marked so in the output for example. Let's see what further investigations will show.

Unfortunately I'm a python noob, otherwise I would offer my help!

[derhuerst]

Another shameless plug: You can always use our public API wrapping the DB HAFAS endpoint. (This of course adds one more centralised component to the system.)

[akloeckner]

I personally think that another server dependency wouldn't be the perfect choice. Another library dependency wouldn't hurt probably.

Assuming DB's API will stay around, I have the impression, your JS implementation of the API is quite sophisticated. So, I wonder wouldn't it be possible to simply wrap it in Python (instead of porting it)? That way, API changes on the DB side would need handling only once.

A quick look around pointed me to JS2PY. It is said to interpret virtually any JS code. What do you think? Schiene3 is coming? ;-)

https://pypi.org/project/Js2Py/

PS: For reference, there is an issue for this subject already: #1
PPS while we're at it. Why not simply include the (Java) implementation of transport-enabler, which you mentioned? We could simply plug into an (apparently?) proven open-source backend...

[derhuerst]

I've proceeded with the discussion in
#1 (comment).