Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Switching search type from TVSEARCH to SEARCH unnecessarily (and only for external requests) #842

Closed
pproba opened this issue Feb 5, 2023 · 12 comments
Closed

[BUG] Switching search type from TVSEARCH to SEARCH unnecessarily (and only for external requests) #842

pproba opened this issue Feb 5, 2023 · 12 comments
Labels
bug

Comments

@pproba
Copy link

pproba commented Feb 5, 2023

I've seen that one indexer (nzbplanet) returned way too many results from a generated query after an external request.
This is the log:

2023-02-05 14:54:50.370  INFO --- [http-nio-0.0.0.0-5] org.nzbhydra.api.ExternalApi             : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Received external newznab API call: NewznabParameters{t=TVSEARCH, q=Formula 1 Drive to Survive, cat=[5000], season=5, offset=0, limit=100, raw=false, o=XML, attrs=[], extended=true, indexers=[]}
2023-02-05 14:54:50.370  INFO --- [http-nio-0.0.0.0-5] org.nzbhydra.api.ExternalApi             : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Executing new search
2023-02-05 14:54:50.370 DEBUG --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Picking indexers out of 5
2023-02-05 14:54:50.373  INFO --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Selected 5 out of 5 indexers: newz-complex, NZB Finder, NZBGeek, nzbplanet, Drunken Slug
2023-02-05 14:54:50.373 DEBUG --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.Searcher          : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Going to call newz-complex, NZB Finder, NZBGeek, nzbplanet, Drunken Slug because their cache is exhausted
2023-02-05 14:54:50.373  INFO --- [pool-5288-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] newz-complex: Calling http://newz-complex.org/www/api?apikey=<apikey>&t=tvsearch&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&password=1&cat=5000&limit=1000&offset=0
2023-02-05 14:54:50.373  INFO --- [pool-5288-thread-2] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZB Finder: Calling https://nzbfinder.ws/api?apikey=<apikey>&t=tvsearch&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&password=1&cat=5000&limit=1000&offset=0
2023-02-05 14:54:50.373 DEBUG --- [pool-5288-thread-3] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZBGeek: Switching search type to SEARCH because this indexer doesn't allow using search type MOVIE/TVSEARCH with a query
2023-02-05 14:54:50.374  INFO --- [pool-5288-thread-3] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZBGeek: Calling https://api.nzbgeek.info/api?apikey=<apikey>&t=search&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&password=1&cat=5000&limit=1000&offset=0
2023-02-05 14:54:50.374  INFO --- [pool-5288-thread-5] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Drunken Slug: Calling https://api.drunkenslug.com/api?apikey=<apikey>&t=tvsearch&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&password=1&cat=5000&limit=1000&offset=0
2023-02-05 14:54:50.374 DEBUG --- [pool-5288-thread-4] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] nzbplanet: Switching search type to SEARCH because this indexer doesn't allow using search type MOVIE/TVSEARCH with a query
2023-02-05 14:54:50.374  INFO --- [pool-5288-thread-4] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] nzbplanet: Calling https://api.nzbplanet.net/api?apikey=<apikey>&t=search&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&password=1&cat=5000&limit=1000&offset=0
2023-02-05 14:54:50.422 DEBUG --- [pool-5288-thread-2] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZB Finder: Found 0 results which were already in the database and 0 new ones
2023-02-05 14:54:50.422  INFO --- [pool-5288-thread-2] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZB Finder: Successfully executed search call in 48ms with 0 total results
2023-02-05 14:54:50.422 DEBUG --- [pool-5288-thread-2] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZB Finder: Returning results 0-0 of 0 available (0 already rejected)
2023-02-05 14:54:50.471 DEBUG --- [pool-5288-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] newz-complex: Found 0 results which were already in the database and 0 new ones
2023-02-05 14:54:50.471  INFO --- [pool-5288-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] newz-complex: Successfully executed search call in 98ms with 0 total results
2023-02-05 14:54:50.471 DEBUG --- [pool-5288-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] newz-complex: Returning results 0-0 of 0 available (0 already rejected)
2023-02-05 14:54:50.501 DEBUG --- [pool-5288-thread-4] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] nzbplanet: Found 0 results which were already in the database and 100 new ones
2023-02-05 14:54:50.504  INFO --- [pool-5288-thread-4] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] nzbplanet: Successfully executed search call in 117ms with 196 total results
2023-02-05 14:54:50.504 DEBUG --- [pool-5288-thread-4] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] nzbplanet: Returning results 0-100 of 196 available (0 already rejected)
2023-02-05 14:54:50.561 DEBUG --- [pool-5288-thread-5] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Drunken Slug: Found 0 results which were already in the database and 0 new ones
2023-02-05 14:54:50.562  INFO --- [pool-5288-thread-5] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Drunken Slug: Successfully executed search call in 187ms with 0 total results
2023-02-05 14:54:50.562 DEBUG --- [pool-5288-thread-5] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Drunken Slug: Returning results 0-0 of 0 available (0 already rejected)
2023-02-05 14:54:52.466 DEBUG --- [pool-5288-thread-3] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZBGeek: Found 0 results which were already in the database and 0 new ones
2023-02-05 14:54:52.466  INFO --- [pool-5288-thread-3] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZBGeek: Successfully executed search call in 2091ms with 0 total results
2023-02-05 14:54:52.466 DEBUG --- [pool-5288-thread-3] org.nzbhydra.indexers.Newznab            : [ID: 32498, Host: 0:0:0:0:0:0:0:1] NZBGeek: Returning results 0-0 of 0 available (0 already rejected)
2023-02-05 14:54:52.466 DEBUG --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.Searcher          : [ID: 32498, Host: 0:0:0:0:0:0:0:1] All indexer caches exhausted
2023-02-05 14:54:52.468 DEBUG --- [http-nio-0.0.0.0-5] o.nzbhydra.searching.DuplicateDetector   : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Duplicate detection for 100 search results found 0 duplicates
2023-02-05 14:54:52.472  INFO --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.Searcher          : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Returning results 1-100 from 100 results in cache. A total of 196 results is available from indexers of which 0 were already rejected and 0 were removed as duplicates
2023-02-05 14:54:52.476  INFO --- [http-nio-0.0.0.0-5] org.nzbhydra.api.ExternalApi             : [ID: 32498, Host: 0:0:0:0:0:0:0:1] Search took 2105ms. Returning 100 results

All indexers except nzbplanet found 0 results because the season hasn't aired yet.
Interestingly, the queries for nzbplanet and NZBGeek have been switched to SEARCH, supposedly because the queries aren't supported by the indexers.

I've tried it out and both nzbplanet and NZBGeek are totally fine with the TVSEARCH query (0 results for season 5 and lots of results for season 4).

nzbplanet returned 196 results because it completely ignores the season attribute if the search type is not TVSEARCH.

It gets more interesting: an internally triggered web-search doesn't change the query type and behaves correctly:

2023-02-05 16:19:43.954  INFO --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.SearchWeb         : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] New search request: SearchRequest{source=INTERNAL, indexers=[nzbplanet], searchType=TVSEARCH, category=TV, offset=0, limit=100, minsize=50, maxsize=5000, identifiers={TVMAZE=41074, TVIMDB=tt8289930, TVDB=359913}, title=Formula 1: Drive to Survive, season=5}
2023-02-05 16:19:43.956 DEBUG --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Picking indexers out of 5
2023-02-05 16:19:43.956  INFO --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Not using newz-complex because it's not in selection [nzbplanet]
2023-02-05 16:19:43.957  INFO --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Not using NZB Finder because it's not in selection [nzbplanet]
2023-02-05 16:19:43.957  INFO --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Not using NZBGeek because it's not in selection [nzbplanet]
2023-02-05 16:19:43.957  INFO --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Not using Drunken Slug because it's not in selection [nzbplanet]
2023-02-05 16:19:43.957  INFO --- [http-nio-0.0.0.0-5] o.n.searching.IndexerForSearchSelector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Selected 1 out of 5 indexers: nzbplanet
2023-02-05 16:19:43.958 DEBUG --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.Searcher          : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Going to call nzbplanet because their cache is exhausted
2023-02-05 16:19:43.959 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.QueryGenerator     : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] No query generation needed for nzbplanet. indexerDoesntSupportRequiredSearchType: false. indexerDoesntSupportAnyOfTheProvidedIds: false. queryGenerationPossible: true. queryGenerationEnabled: true. fallbackRequested: false
2023-02-05 16:19:43.959  INFO --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Calling https://api.nzbplanet.net/api?apikey=<apikey>&t=tvsearch&extended=1&tvmazeid=41074&tvdbid=359913&season=5&minsize=50&password=1&cat=5000&limit=1000&offset=0
2023-02-05 16:19:44.153 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Found 0 results which were already in the database and 0 new ones
2023-02-05 16:19:44.153  INFO --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Successfully executed search call in 192ms with 0 total results
2023-02-05 16:19:44.153 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Returning results 0-0 of 0 available (0 already rejected)
2023-02-05 16:19:44.153  INFO --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: No results found for ID based search. Will do a fallback search using a generated query
2023-02-05 16:19:44.153 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.QueryGenerator     : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Removed illegal characters from title 'Formula 1: Drive to Survive'. Title that will be used for query is 'Formula 1 Drive to Survive'
2023-02-05 16:19:44.153 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.QueryGenerator     : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Search request provided title Formula 1 Drive to Survive. Using that as query base.
2023-02-05 16:19:44.154 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.QueryGenerator     : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Indexer does not support any of the supplied IDs or the requested search type. The following query was generated: Formula 1 Drive to Survive
2023-02-05 16:19:44.154  INFO --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Calling https://api.nzbplanet.net/api?apikey=<apikey>&t=tvsearch&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&minsize=50&password=1&cat=5000&limit=1000&offset=0
2023-02-05 16:19:44.250 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Found 0 results which were already in the database and 0 new ones
2023-02-05 16:19:44.250  INFO --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Successfully executed search call in 96ms with 0 total results
2023-02-05 16:19:44.251 DEBUG --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab            : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Returning results 0-0 of 0 available (0 already rejected)
2023-02-05 16:19:44.252 DEBUG --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.Searcher          : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] All indexer caches exhausted
2023-02-05 16:19:44.253 DEBUG --- [http-nio-0.0.0.0-5] o.nzbhydra.searching.DuplicateDetector   : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Duplicate detection for 0 search results found 0 duplicates
2023-02-05 16:19:44.254 DEBUG --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.Searcher          : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Will load all cached results
2023-02-05 16:19:44.260  INFO --- [http-nio-0.0.0.0-5] org.nzbhydra.searching.SearchWeb         : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] Web search took 305ms

This only shows the log for nzbplanet but the same is happening for NZBGeek.
@pproba pproba added the bug label Feb 5, 2023
@theotherp
Copy link
Owner

Please create and post the debug infos zip.

@pproba
Copy link
Author

pproba commented Feb 5, 2023

nzbhydra-debuginfos-2023-02-05-17-30.zip

@theotherp
Copy link
Owner

In 2021 NZBGeek as well as nzbplanet did not support t=tvsearch without ID keys (e.g. tvdb ID). That seems to have changed. I can only verify for nzbgeek but if you tested it with nzbplanet I'll take your word for it. So to be clear, /api?t=search&apikey=APIKEY&extended=1&q=Formula 1 Drive to Survive on nzbplanet works?

The internal search does not change the type because you can see that using the autocomplete IDs were provided (identifiers={TVMAZE=41074, TVIMDB=tt8289930, TVDB=359913})

@pproba
Copy link
Author

pproba commented Feb 5, 2023 •
edited
Loading

nzbplanet:
/api?apikey=<apikey>&t=tvsearch&q=Formula%201%20Drive%20to%20Survive&season=1&extended=1&password=1&cat=5000&limit=1000&offset=0 -> <newznab:response offset="0" total="33"/>

/api?apikey=<apikey>&t=search&q=Formula%201%20Drive%20to%20Survive&season=1&extended=1&password=1&cat=5000&limit=1000&offset=0 -> <newznab:response offset="0" total="196"/> (results contain episodes from all seasons)

Regarding the internal search, I don't fully understand. Why did hydra send this query:
2023-02-05 16:19:44.154 INFO --- [pool-5358-thread-1] org.nzbhydra.indexers.Newznab : [ID: 55558, Host: 2003:d0:3f46:1800:1c4a:e0dd:20f:222] nzbplanet: Calling https://api.nzbplanet.net/api?apikey=<apikey>&t=tvsearch&extended=1&q=Formula%201%20Drive%20to%20Survive&season=5&minsize=50&password=1&cat=5000&limit=1000&offset=0
The mentioned tvmaze, tvimdb and tvdb ids were not part of the query. This is the title based fallback path.

@theotherp
Copy link
Owner

Sorry, that was wrong.

Previously nzbplanet did not support t=tvsearch without ID keys so hydra switched to t=search (search ID 32498 in the log).

For search ID 55558 in the log , that's apparently a bug. When a fallback query is requested the fact that tvsearch is (supposedly) not supported with a query is ignored. But as the indexers all seem to support that this bug is fixed as well.

@pproba
Copy link
Author

pproba commented Feb 5, 2023

Makes sense, thanks for looking into this!

@reloxx13
Copy link

reloxx13 commented Feb 6, 2023

is this hardcoded?

There is a capabiliy check which should determine if it is available or not.

grafik

@theotherp
Copy link
Owner

theotherp commented Feb 6, 2023 •
edited
Loading

Is what hardcoded?

The way the API is implemented by indexers is all over the place. An indexer may support TVSEARCH but only when used in combination with IDs or not support season/episode parameters for regular searches or whatever.

Previously those indexers mentioned above allowed TVSEARCH when used in combination with an ID but not when used in combination with a query. They seem to support that now which is why I removed the extra handling for them.

@reloxx13
Copy link

reloxx13 commented Feb 6, 2023

Is what hardcoded?

the capabilities of the indexer or why do you have to change the searchtype for a special indexer urself.

@theotherp
Copy link
Owner

Yes, that special part was hardcoded. There are so many special cases that I don't want to check for them each time.

@reloxx13
Copy link

reloxx13 commented Feb 6, 2023

But it should not be used when the user can use the capability check :/

@pproba
Copy link
Author

pproba commented Feb 6, 2023

The way the API is implemented by indexers is all over the place. An indexer may support TVSEARCH but only when used in combination with IDs or not support season/episode parameters for regular searches or whatever.

This info is part of the capabilities though. I'm starting to agree with @reloxx13, there shouldn't be a need for hardcoding indexer-specific behavior. That being said, I have no idea how much time it would take to implement this instead of changing some hard-coded values every once in a while.

nzbgeek:

<searching>
  <search available="yes" supportedParams="q,group"/>
  <tv-search available="yes" supportedParams="q,rid,tvdbid,season,ep"/>
  <movie-search available="yes" supportedParams="q,imdbid,genre"/>
  <audio-search available="yes" supportedParams="q"/>
</searching>

nzbplanet:

<searching>
  <search available="yes" supportedParams="q"/>
  <tv-search available="yes" supportedParams="q,rid,tvdbid,tvmazeid,season,ep"/>
  <movie-search available="yes" supportedParams="q,imdbid,genre"/>
</searching>

Both list "q" as supported parameter in tv-search queries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pproba @theotherp @reloxx13