Optional search restriction in epgsearch

[prl001]

There have been a number of requests on the Beyonwiz forum to allow epgsearch to restrict its search to the current bouquet, especially for users who normally have "Enable multiple bouquets" disabled.

I intend making the search choices "whole EPG/current bouquet/current channel".

I'm happy to implement this, but I have a question about what the default setting should be. For user convenience, I'd prefer the default to be "current bouquet". Backwards compatibility would argue for "whole EPG".

Does anyone (especially the oe-alliance reviewers) have opinions either way on this? Or even whether it's a good idea at all?

[RobvanderDoes]

1- Where is this search-choice being located (in the GUI I mean)?
2- Please let 'all bouquets' be the default value.

[RobvanderDoes]

And speaking for myself: sometimes I would like to search only in the active service, an other time in the active bouquet and sometimes in all bouquets. So maybe on hitting 'search' the question service/bouquet/all should arise?
And then it would be a good idea indeed 👍

[arn354]

add a epgsearch config option config.plugins.epgsearch.mode (all, current bouquet, current service)
use boxbranding to set default config.plugins.epgsearch.mode for beyonwiz to current bouquet (else "all").
The default behaviour should not change in my opinion.

[RobvanderDoes]

Then I would need to enter the setup every single time I want an other search 👎

As I said: the question where to search on every search would be ideal IMHO.

[prl001]

How about a config.plugins.epgsearch.defaultmode setting instead to set the search mode popup's default? That way you can choose whatever search you want at search time, but have your most common search directly on OK.

I like the idea of using boxbranding to select a different setting default on the Beyonwiz, if I want to change the Beyonwiz default.

[arn354]

How about a config.plugins.epgsearch.defaultmode setting instead to set the search mode popup's default?
using boxbranding to select a different setting default on the Beyonwiz

Thats what i meant.

[prl001]

But perhaps better might be:
Search scope: All bouquets/Current bouquet/Current service/Ask
Default search scope: All bouquets/Current bouquet/Current service

where "Default search scope" is only visible and active when "Search scope" is "Ask", and it preselects which popup item has focus, so that if you have "Ask" set, YELLOW, OK always does your favourite search.

That way, people who want to be able to chose a fixed scope have no popup and people who want a popup can have the popup focus preselect to their usual search.

[arn354]

Good point

[RobvanderDoes]

I was about to suggest that; sounds good to me.

[AbuBaniaz]

A bit unrelated to the topic, but as you are looking at EPG search, I might as well as ask now.

Can you add the orbital position in the result too? This is very handy for users with the multiple tuners. One of our members has the same services on 28.2E, 9E and DVB-T. Having these will make it easy to set a timer on a specific tuner.

On OpenVix we added it to History Zap.
OpenViX/enigma2@cc0b141

I attach a picture of how history zap looks like

[RobvanderDoes]

Would be great if orbit pos could be added!

[prl001]

Unfortunately, I have no access to an enigma2-based satellite PVR, so I would have no way to test adding orbital position to the search results. The Beyonwiz PVRs I have access to only have DVB-T tuners, and I have no satellite subscription.

[RobvanderDoes]

Could I be of help? Either by testing or allowing you entry to a box?

[prl001]

Thanks, Rob.

Having a tester might be enough, but I'd need to have a quick look at the code to decide whether I can fake up the orbital position on a DVB-T service for my own testing before giving the code to someone to test live.

That will have to wait til tomorrow. It's bedtime here in Australia :)

[RobvanderDoes]

OK, then being in contact via Skype seems the way to go (robvanderdoes). I'm an early riser in Europe, so we might have some overlapping working time :)

[Huevos]

1, current default is all services, i.e.includes services not in a bouquet. So options would be, all services, services fron all bouquets, services from current bouquet, current service.

2, orbital position can get extracted from service ref. Same for terrestrial and cable.

[prl001]

So options would be, all services, services fron all bouquets, services from current bouquet, current service.

That can be done.

I've had a look at the orbital position code in OpenViX/enigma2, which seems to be pretty much a copy/paste of the code in Components/TimerList.py in the Beyonwiz enigma2. So that should be easy.

Rob, I'll get back to you to arrange a contact time when I have something worth looking at.

[Huevos]

Was not copy and paste code.

[prl001]

OK. "well established code pattern", then :)

[prl001]

EPGSearchSetup writes setting description texts into a Components.Sources.StaticText widget named "status". The convention for the Beyonwiz enigma2 description text widget name is "description". The "status" widget doesn't seem to have any function.

I can make the description work just by adding a Label() widget in self["description"].

Should I get rid of EPGSearchSetup.self["status"] or leave it as it is for backwards compatibility?

[RobvanderDoes]

'Status' has been used for [Recording/Timeshift] modify code to use main setup.xml file. ViX doesn't use this any more (old Recording_timeshift file), but other teams might.
See OpenViX/enigma2@3177f5e#diff-d4e91cd32603b597a4b04dff33cc86a2L146
It seems safe to remove. And if you do that in a separate commit it will be easy to revert when required.

[prl001]

OK, I'll do it that way, then.

[prl001]

OK, if the search scope is "ask user", should it ask only on entry to the EPG Search screen, or on each search, e.g. from YELLOW New Search, or from MENU>Show search history?

I.e. should "ask user" be per session or per query?

I've already got the mechanism in place for remembering the last answer to "ask user", it's needed at times like when the callback from timerAdd() calls refreshlist().

[RobvanderDoes]

For me the setting 'ask user' should be applicable to all possible searches.

[prl001]

Good. That's how it is now :)

[prl001]

I've created a pull request for the changes that add search scope specification (all services/all bouquets/current bouquet/current service/ask user) to EPG Search.

As a bonus, I've thrown in a shortcut to EPG search history on long-YELLOW.

I'll do the request for orbit position display in the search results as a separate pull request, because it's really unrelated to the search scope.

[RobvanderDoes]

Thanks, I'm about to test it.
And indeed: orbit-pos should be in a different commit.
And what about the 'status' widget?

[prl001]

And what about the 'status' widget?

Taken out in a commit of its own in the search scope pull request.

[prl001]

Rob has merged my pull request to add search scope to EPG Search. Unfortunately, it still has the buggy code in it that causes the crash on exit from EPG Search Setup on (at least) VIX. Please don't use it yet.

[RobvanderDoes]

Well, sorry, there were quite a few recent requests, so I thought all was in already.
Anyway: you have the fix ready :)

[RobvanderDoes]

And when changing the name via the GUI and changing it back to the original name, the name entry in the userbouquet remains in place:
#SERVICE 1:0:1:3616:C8:13E:820000:0:0:0::ESP2 German
#DESCRIPTION ESP2 German

[prl001]

And when changing the name via the GUI and changing it back to the original name, the name entry in the userbouquet remains in place:

Yes, same on the Beyonwiz image.

[Sayyid-A]

And when changing the name via the GUI and changing it back to the original name, the name entry in the userbouquet remains in place:
#SERVICE 1:0:1:3616:C8:13E:820000:0:0:0::ESP2 German
#DESCRIPTION ESP2 German

This does not make sense, it should reset!
#SERVICE 1:0:1:3616:C8:13E:820000:0:0:0:

BTW, this is how it works in E-Channelizer.

[RobvanderDoes]

Well, the fact is that the STB's GUI has no 'reset' option. So one has to use 'rename' in order to get the original name back in (provided one remembers what that name is).

[Huevos]

Has the current version you are testing been committed yet?

[RobvanderDoes]

No.

[prl001]

And it's not yet ready to commit.

[prl001]

This is what I hope will be the last test version before I post a pull request.

The handling of user-assigned service names is:

• If the user-assigned name in the bouquet serviceref and the name for the service in lamedb are the same, only one result will be returned for each event on that serviceref.
• If there are user-assigned names that are different from the name for the service in lamedb, then they will be treated as though they are different services in doing search matches.

So, if lamedb has:

0211:eeee0000:0211:1010:1:0
ABC
p:ABC

and the bouquets being searched contain:

1:0:1:211:211:1010:EEEE0000:0:0:0:
1:0:1:211:211:1010:EEEE0000:0:0:0::ABC
1:0:1:211:211:1010:EEEE0000:0:0:0::ABC SD

Searches that include lamedb ("all services") implicitly include a search for the first form.

Each event on serviceref 1:0:1:211:211:1010:EEEE0000:0:0:0: matching the search string will return:
Mon 1/1, 12:34 ABC: ABC News at Noon
Mon 1/1, 12:34 ABC SD: ABC News at Noon

If a user specifies more than one name for the same service, there doesn't seem to be any sensible way of determining which one should be returned in the search, so they are treated as though they are different services in the search results.

I've also done some work on speeding up match processing when the search returns a large number of matches, and when the set of services being searched is small, the search goes back to using the old per-service EPG contents list with the name matching in Python, rather than the search of the whole EPG on the name and then filtering by serviceref.

I hope that this will be the last test version. I'm interested in hearing if there are any anomalies in the amount of time that a search takes, and the circumstances that caused the anomaly. Anything that takes more than about 3 seconds should be reported, though personally I think it would be best if the function always took less than 1 sec. However, if the number of entries in the EPG is vary large, there's not much than can be done about the underlying search without changing the C++ API to eEPGCache.

I'd also like to hear about any anomalies in the results returned, like duplicated search results where the result should not be duplicated.

There's still some execution time reporting in the debug code in this, so if you do get a search that takes too long, please post a copy of a debug log for the search.

This test version of EPGSearch.py should be used with the current version of EPGSearch in the repository.

EPGSearch.zip

[RobvanderDoes]

All searches I did (except current service) took quite a long time. Please have a look at the log.
Enigma2_debug_2016-08-26_06-26-03.zip

[prl001]

Thanks. Not what I wanted to hear, though.

I can see where the problem area is, and I can simulate the behaviour locally.

[hd75hd]

Great
I have no duplicate results anymore

Thanks !

[prl001]

Good that that bit is working properly!

[prl001]

OK. Another test version. The previous one had some debug prints in an inner loop of the service filter setup that caused about 12 sec delay in the "all services"/"all bouquets" tests that Rob ran.

They've been taken out and I've done some further optimisation on the filter setup code and sped that up (over the previous code without the debug prints) by about 30%.

That still means that Rob's test searches on "all services"/"all bouquets" are likely to still take about 5-6 sec, nearly 2 sec for the filter setup and nearly 4 sec for the actual search of the EPG data in the C++ code. Rob's "current bouquet" search will probably take about 4sec, with the time dominated by the low-level search.

I've tried to reduce the "I pressed the button but nothing happened" feel of the search by displaying "Searching..." text while the search is running. It doesn't help with the speed, but at least you know that something's happening.

I tried several options to try to get the filter setup time down, but, as is usual with optimisation, the speedups were getting smaller and smaller with each round and the later rounds couldn't justify making the code less readable (I'm talking about < 100ms improvements over a total of 4-6 sec).

There are still timing debug prints in the code so if my forecasts of the likely performance on Rob's tests are over-optimistic, I can still have a look at where the time is going.

I'm disappointed that I can't get these searches to run faster on typical satellite setups.

EPGSearch.zip

[RobvanderDoes]

Not bad at all. The 'searching' is indeed a (cosmetic) improvement, and speed has increased quite a bit. Attached two sets of logs; one is comparable with my previous tests (same box, same setup, same amount of EPG-data), the other is from my main-box (no remotes, more EPG-data).
IMHO this is all quite acceptable.

Downloads.zip

So unless other testers would chip in with quite different results, I would say: time for a pull request 👍

[prl001]

Thanks Rob. The times are in line with what I'd expect them to be. If there are no unexpected problems, I'll do a pull request tomorrow my time.

I'm still disappointed that it's not faster.

[RobvanderDoes]

Good: awaiting your bloody final last pullrequest :)

In the meantime I'll try it on a 400Mh box, to see if this works fine for the elder generation as well.

[RobvanderDoes]

Done the latter (on the same box I previously reported about, with the extreme long times (VU Ultimo)): considerably longer, but IMHO not unacceptable. In fact only the 'all services' is taking too long, 'all bouquets' isn't too bad. See log.
Enigma2_debug_2016-08-27_13-23-22.zip
But maybe others think otherwise?

[prl001]

In fact only the 'all services' is taking too long, 'all bouquets' isn't too bad. See log.

In that setup, "all bouquets" only has around 3000 filter entries, while "all services" about 8000, and the filter preparation times are respectively ~8 and ~15 sec. The EPG must only have relatively few entries, because the low-level name search over the EPG is only taking about 0.6 sec, even though the box is quite a lot slower than in the other tests.

[prl001]

I've created a pull request for these changes.

[RobvanderDoes]

Thanks for all your work and the cooperation.

[prl001]

No problem.

[prl001]

And thanks for your, and the others' help in testing

[RobvanderDoes]

Seeing the progress and the bug-fixing testing was actually a pleasure :)

Which plugin is next?

[prl001]

Which plugin is next?

IceTV. 😉

[RobvanderDoes]

That sounds like not much testing for me then.
But it would be good to add it to (or incorporate it in one of the existing) OE-A-plugins, as we do indeed have Australian users of several images.

And if you're good at embedding EPG-readers I have something more in mind for you....

[prl001]

I didn't write the IceTV plugin. It was written (under contract, I believe) for Beyonwiz Australia, and its copyright notice makes it clear it's not Open Source.

I'm not sure what the ramifications of using it in other images would be. It may require IceTV to OK the plugin for use with other devices and to provide a suitable API key. IceTV provides support services for subscribers and they may not wish to take on other devices.

[hd75hd]

I also want to thank you for your hard work on this Plugin

[prl001]

No problem. Thanks for your suggestions and help in testing it.

[prl001]

Thanks to everyone for suggestions and their help in testing. I'm assuming from the silence that there aren't any more major issues.