Unexpected error in horizon.tv thread

[easdue]

The last couple of days I've noticed that tv_grab_nl3.py keeps running using 100% cpu. After running it manually I noticed the following error being thrown:

An unexpected error has occured in the horizon.tv thread:
While fetching the base pages
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1996, in run
self.init_channel_source_ids()
File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 2143, in init_channel_source_ids
self.source_data[ptype]['url'] = self.source_data[ptype]['alt-url']
KeyError: u'channel'

I've attached the request conf an log files
tv_grab_nl3.tar.gz

[hikavdh]

Try deleting the source definition files in ~/.xmltv/sources (in the userdirectory of the user running it!), so they get downloaded afresh.

[easdue]

Tried that but the same error happens. I also tried --disable-source 5 where 5 is the horizon.tv source but that also gives the same error and in all cases tv_grab_nl3.py keeps running and using 100% cpu

…

On donderdag 8 maart 2018 15:52:47 CET Hika van den Hoven wrote: Try deleting the source definition files in ~/.xmltv/sources (in the userdirectory of the user running it!), so they get downloaded afresh.

[hikavdh]

I found the cause for the KeyError and am mostly surprised I have never seen it before. It comes from an error in tv_grab_API.json I just fixed. It might take ten minutes or so to propagate through any cashed data between you and github.
The weird thing is that this code bit deals with switching between the regularly alternating urls for horizon and should have shown up long time ago. Let me know if this solves it.

[hikavdh]

One other question. Does that high load stay indefinitely until you kill the script, and by indefinitely I mean more then 30 minutes?

[easdue]

Yes it keeps running indefenitely using 100% cpu. The first time I noticed it, it was launched at 00:10 (anacron) and I killed it in the afternoon.

…

On vrijdag 9 maart 2018 00:37:44 CET Hika van den Hoven wrote: One other question. Does that high load stay indefinitely until you kill the script, and by indefinitely I mean more then 30 minutes?

[easdue]

Just ran tv_grab_nl3.py but unfortunately it still gives the same error

…

On vrijdag 9 maart 2018 00:28:04 CET Hika van den Hoven wrote: I found the cause for the KeyError and am mostly surprised I have never seen it before. It comes from an error in tv_grab_API.json I just fixed. It might take ten minutes or so to propagate through any cashed data between you and github. The weird thing is that this code bit deals with switching between the regularly alternating urls for horizon and should have shown up long time ago. Let me know if this solves it.

[hikavdh]

The same KeyError: u'channel'?
That can't be as that key is no longer in data_def_names["source"]. I changed it to the correct u'channels'.
And tv_grab_API.json is always downloaded from git. So send me a log with the current error.

[easdue]

I think you forgot to push tv_grab_API.json to github. The version there is 8 month old

…

On vrijdag 9 maart 2018 21:27:46 CET Hika van den Hoven wrote: The same `KeyError: u'channel'`? That can't be as that key is no longer in data_def_names["source"]. I changed it to the correct u'channels'. And tv_grab_API.json is always downloaded from git. So send me a log with the current error.

[hikavdh]

I guess you're referring to the dummy copy inside the package. I only update that on a new release. The live one is in the sourcematching repository.

[easdue]

Sorry for that, it now complains about base-channels Loaded the English texts file Using config file: /var/lib/tvheadend//.xmltv/tv_grab_nl3_py.conf An unexpected error has occured in the horizon.tv thread: While fetching the base pages Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 1996, in run self.init_channel_source_ids() File "/usr/lib/python2.7/site-packages/tvgrabpyAPI/tv_grab_fetch.py", line 2143, in init_channel_source_ids self.source_data[ptype]['url'] = self.source_data[ptype]['alt-url'] KeyError: u'base-channels' [tv_grab.zip](https://github.com/tvgrabbers/tvgrabpyAPI/files/1799228/tv_grab.zip)

On vrijdag 9 maart 2018 21:27:46 CET Hika van den Hoven wrote: The same `KeyError: u'channel'`? That can't be as that key is no longer in

data_def_names["source"]. I

changed it to the correct u'channels'. And tv_grab_API.json is

always

…

downloaded from git. So send me a log with the current error.

[hikavdh]

OK, I get what is happening and to properly fix it, it needs some code adaption. But I think I can fix it for now by adding dummy entries to the horizon definition. I have to check if that won't have any repercussions else. The trouble is that this happens before the thread is tagged active, so the thread manager is not aware and does not correct the state resulting in an indefinite lock-up.

[hikavdh]

That's also why disabling the source does not help.

[hikavdh]

What puzzles me however is that it has not come up before and does not on for instance my system. This part is in place for over a year.

[hikavdh]

That won't do. The dummy entries get filtered by the conversion. I'll see if I can push a fix sometime soon.

[erikdrenth]

Thanks for being on it. I'm having the same issues as easdue with exactly the same error (line 1996 and 2143). Can send more info if needed

[easdue]

I'm running fedora 27 and I have been using tvgrabpy since this summer. The error popped up I think last week so maybe an update here or there is the culprit (fedora is bleeding edge)

[hikavdh]

Try https://github.com/tvgrabbers/tvgrabpyAPI/releases/tag/beta-1.0.8-20180310

[easdue]

Yep, That seems to have solved the problem Thanks Hika

…

On zaterdag 10 maart 2018 19:10:35 CET Hika van den Hoven wrote: Try https://github.com/tvgrabbers/tvgrabpyAPI/releases/tag/beta-1.0.8-20180310