Improve performance of mythtv-setup "Fetch channels from listing source" for xmltv grabbers

[garybuhrmaster]

Improve performance of mythtv-setup "Fetch channels from listing source" for xmltv grabbers

This benefits the new user first experience improvement during mythtv-setup who also have large lineups (poster child would be US cable customers with a large number of channels)

With support in mythfilldatabase for grabbers preferred technique of grabbing allatonce, the "Fetch channels from listing source" request in videosources has performed slowly for grabbers that support allatonce grabbing that have a large list of channels and schedule/program data due to mythfilldatabase doing an allatonce grab even when mythfilldatabase is called with the --only-update-channels option, and processing the guide data even when called with that same option. This is seen most noticeably with US cable/satellite systems (which may have many hundreds of channels and hundreds of thousands of program schedules for the entire dataset).

There are a couple of ways to improve the elapsed time when mythfilldatabase is requested to do --only-update-channels:

• Do not process guide data when --only-update-channels is specified.
• Fall back to the XMLTV baseline capability of day-by-day grabbing and only process one day of schedule/program data rather than allatonce. (Ideally one would even use a "--days 0" specification for the grabber for this case, but not all grabbers reportedly handle --days 0 properly, so one is forced to use the minimum of one day, as the code currently does).
• For grabbers that support --list-channels, use that option to return just the channels (this approach was opened in MythTV trac ticket 12727 (depends on 12706), both tickets were closed as part of the great ticket purge, so are logically being reintroduced here).

Each of these approaches improves performance in different ways for different grabbers depending on their specific capabilities (one size (patch) does not fit all grabbers due to their different capabilities, and only a few grabbers will typically be used with the fetch channels option)

A pull request will be forthcoming for review by the developers.

For a rough estimate of experience improvements using the same large US cable provider (~640 stations, which (due to cable channel number duplication) end up being around ~920 channels, which has ~260000 program schedule entries which end up loading ~370000 programs (multiple channels with the same xmltvid)), with the same xmltv grabber that supports allatonce, running on an older (slow) development system (so the absolute numbers should not be treated as accurate, and the times include (especially for the allatonce logic) obtaining and generating the guide data from the upstream source(s), but they do provide some useful relative comparisons).

• ~45 minutes elapsed for current logic (allatonce grab with loading all guide data into the db)
• ~8 minutes elapsed with not loading any guide data (allatonce grab still creates a file with all the guide data, but it is not loaded into the db)
• ~3 minutes elapsed when bypassing allatonce (force one day grab for --only-update-channels but still loading that one days guide data into the db)
• ~1 minute elapsed with both previous patches applied (one day grab without loading guide data into the db)
• ~8 seconds elapsed when using --list-channels (one only gets the channels, bypassing creating the guide data entirely)

It should be acknowledged that after running mythtv-setup and creating new videosources and input connections that one is expected (and prompted) to run mythfilldatabase which for these same large guide sources will take some time (the ~45 minutes for the first run in the experience numbers previously noted); that is not a new thing, but, significantly, it will no longer be part of initial setup (to be fair, if the data is loaded during the initial fetch channels in mythtv-setup, the first run of mythfilldatabase will be typically faster since at least the majority of the program data is already loaded, so the reduction in database changes will result in elapsed time improvements for the first mythfilldatabase that will no longer be seen).

It should also be noted (that as a core developer previously mentioned on the developement list) that using a grabbers --get-lineup capabilities (where supported (this patch set detects the needed capability, so would presumably be a prereq)) can result in much better channel definitions in addition to fast loading, but that effort is beyond the current issue scope (currently --get-lineup loading of channels into MythTV is supported by a 3rd party tool using the services API as described on the wiki to maintain the improved channel definitions).

Thanks for your consideration of this issue and (eventually) pull request.

[garybuhrmaster]

This was resolved by PR #413 being merged. Thanks.