[rarbg] always have issues searching after a while

[hellcry37]

Please use the search bar at the top of the page and make sure you are not creating an already submitted issue.
Check closed issues as well, because your issue may have already been fixed.

Please read our Contributing Guidelines before submitting your issue to ensure a prompt response to your bug.

Environment

OS: docker container

.Net Runtime: [.Net-Core/.Net-Framework/Mono]

.Net Version:

Jackett Version: v0.17.611

Last Working Jackett Version: n/a

Are you using a proxy or VPN? [yes/no] NO

Description

[List steps to reproduce the error and details on what happens and what you expected to happen]
I am using radarr with jackett indexers, the issue that i have is like in the fallowing example:
Radarr add new movie: V for Vendetta
Radarr search manually, rarbg indexer founds 0 results
I go search the rarbg site, movie is there.
I go search in jackett manually selecting: manual search -> query V for Vendetta -> tracker RARBG -> category Any
results: Manual search in rarbg for V for Vendetta => Found 0 releases
I go search in jackett manually selecting: manual search -> query Vendetta -> tracker RARBG -> category Any
results: Manual search in rarbg for Vendetta => Found 100 releases
I go search in jackett manually selecting: manual search -> query for Vendetta -> tracker RARBG -> category Any
results: Manual search in rarbg for for Vendetta => Found 15 releases
Radarr will never search for last two texts, from here will never find the movie on RARBG

I am using linuxserver/jackett:latest container.
environment:
- "TZ=Europe/Bucharest"
- AUTO_UPDATE=true

Note:
After I restart docker container that runs Jackett it works:

Searching for the same movie V for Vendetta from Radarr gives this log:

2021-03-03 12:16:51	Info	Torznab search in RARBG => Found 7 releases
2021-03-03 12:16:49	Info	Torznab search in DataScene => Found 1 releases
2021-03-03 12:16:48	Info	Torznab search in SpeedApp => Found 3 releases
2021-03-03 12:16:02	Info	Jackett startup finished in 7.502 s

Note 2:
I forgot to mention this behavior happens from a long time already, by that I mean when I search movies in Radarr, RARBG indexer just don't work anymore. Problem is always fixed after a container restart.

Logged Error Messages

2021-03-03 12:02:41	Info	Torznab search in RARBG for V de la vendeta 2006 => Found 0 releases (from cache)
2021-03-03 12:02:39	Info	Torznab search in RARBG for V for Vendetta 2006 => Found 0 releases (from cache)
2021-03-03 12:02:37	Info	Torznab search in DataScene => Found 1 releases (from cache)
2021-03-03 12:02:35	Info	Torznab search in RARBG => Found 0 releases (from cache)
2021-03-03 12:02:33	Info	Torznab search in SpeedApp => Found 3 releases (from cache)
2021-03-03 11:59:46	Info	Manual search in rarbg for for Vendetta => Found 15 releases
2021-03-03 11:59:37	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:36	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:34	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:22	Info	Manual search in rarbg for Vendetta => Found 100 releases
2021-03-03 11:59:18	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:18	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:17	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:16	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:16	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:16	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:15	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:15	Info	Manual search in rarbg for V for Vendetta => Found 0 releases (from cache)
2021-03-03 11:59:12	Info	Manual search in rarbg for V for Vendetta => Found 0 releases
2021-03-03 11:58:40	Info	Jackett is already updated. Current version: v0.17.611
2021-03-03 11:58:40	Info	Checking for updates... Jackett variant: CoreLinuxAmdx64
2021-03-03 11:58:35	Info	Test search in The Pirate Bay => Found 50 releases
2021-03-03 11:58:35	Info	Test search in DataScene => Found 100 releases
2021-03-03 11:58:34	Info	Test search in FileList => Found 100 releases
2021-03-03 11:58:34	Info	Test search in SpeedApp => Found 100 releases
2021-03-03 11:58:34	Info	Test search in RARBG => Found 100 releases

Screenshots

[Place any screenshots of the issue here if needed]

[ilike2burnthing]

Radarr and Sonarr have their own RARBG indexer, so it may just be easier to use it.

You're getting repeated instances of 0 results because it's being fetched from Jackett's cache, and for whatever reason it found 0 results the first time (by default the cache clears after 35mins).

RARBG's API regularly fails, there's nothing we can do about that.

I get 10 results for the film when using Radarr. It uses the IMDB ID (tt0434409) rather than the name. I get the same 10 results when searching tt0434409 in Jackett.

Hope that helps explain what you're seeing.

[hellcry37]

sonarr and radarr RARBG indexers fails constantly, in my opinion jackett rarbg indexer works way better

[garfield69]

@6cUbi57z could you add the polly retry support to the rarbg indexer?
this api server often has intermittent failures during queries, and having another go soon after often works.

[6cUbi57z]

Can do but it might be worth considering a better way to implement it. Either something on one of the base indexer classes or on the calling code. There would also be the question of if it should be a global setting or a setting for each indexer?

I'll add it to RARBG on my PR for now for a quick fix. If we think it should go on more indexers I would suggest a new issue.

[6cUbi57z]

@garfield69 Looks like the RarBG indexer already uses a retry which is on the BaseWebIndexer class. It's implemented manually rather than using polly and not configurable.

Jackett/src/Jackett.Common/Indexers/RarBG.cs

Lines 135 to 143 in 8c0fcec

	private async Task<IEnumerable<ReleaseInfo>> PerformQueryWithRetry(TorznabQuery query, bool retry) {
	var releases = new List<ReleaseInfo>();
	// check the token and renewal if necessary
	await RenewalTokenAsync();
	var response = await RequestWithCookiesAndRetryAsync(BuildSearchUrl(query));
	var jsonContent = JObject.Parse(response.ContentString);
	var errorCode = jsonContent.Value<int>("error_code");

Jackett/src/Jackett.Common/Indexers/BaseIndexer.cs

Lines 447 to 471 in 8c0fcec

	protected async Task<WebResult> RequestWithCookiesAndRetryAsync(
	string url, string cookieOverride = null, RequestType method = RequestType.GET,
	string referer = null, IEnumerable<KeyValuePair<string, string>> data = null,
	Dictionary<string, string> headers = null, string rawbody = null, bool? emulateBrowser = null)
	{
	Exception lastException = null;
	for (var i = 0; i < 3; i++)
	{
	try
	{
	return await RequestWithCookiesAsync(
	url, cookieOverride, method, referer, data, headers, rawbody, emulateBrowser);
	}
	catch (Exception e)
	{
	logger.Error(
	e, string.Format("On attempt {0} downloading from {1}: {2}", (i + 1), DisplayName, e.Message));
	lastException = e;
	}
	await Task.Delay(500);
	}
	throw lastException;
	}

I guess if it's already in the base class, there's argument for changing PR #11318 and just making it configurable on the base class.

Next question would be, should it be configurable for each indexer or as a global setting? My preference would be per indexer as it can slow down API requests and I would prefer to only add retries for problematic sites.

[hellcry37]

great news. i am glad someone trys to improve this. thank you

[garfield69]

re: rarbg, ah well, if there is already a retry for this and it still fails regularly, then its clearly a problem with the rarbg API server and there is nothing more that we can do from a Jackett POV.
re: adding polly to base: not in favour of this. We add it to an indexer in the rare instance we think it will be helpful, but not as a matter of course. Web sites that have dodgy servers are not Jackett's problem. The mandate is not to provide an indexer at all costs, just on a best effort basis ;-)

[6cUbi57z]

re: adding polly to base: not in favour of this. We add it to an indexer in the rare instance we think it will be helpful, but not as a matter of course. Web sites that have dodgy servers are not Jackett's problem. The mandate is not to provide an indexer at all costs, just on a best effort basis ;-)

Completely understand this. I was hesitant to add it to the Anidex indexer but without any influence over the Anidex site it's difficult to come up with an alternative and beats going into Sonarr and hitting the test button every time it fails (especially when Sonarr makes multiple calls per search). Jackett is already acting as an abstraction layer so adding some retry logic seemed the nicest solution without adding another app or proxy.

My thoughts with putting polly the base class were more around consistency. If PR #11318 is approved there will be 2 separate implementations of retry logic. It would also add flexibility to allow the user to change the retry settings without needing code changes (default to no retry) or with only a minor code change (property on the indexer classes which can be overridden to make it configurable?)

Up to you. I'm happy to take on the change but if you don't think it's right I'll leave it be.

[garfield69]

So let me get this straight in my head.
You are proposing to add polly retry code to the base, but with the default set to no-retry.
And then add to anidex and rarbg an override to default to 1 retry, and optionally a indexer config selector to allow users of the anidex and rarbg the option to increase the retry count?
If the above is what you intended, then that seems like a reasonable approach which I would approve.

[6cUbi57z]

@garfield69 That sounds right. I've updated my merge request and put more details on there. I expect this may go back and forth a bit so lets move the discussion to the PR.