TorrentAPI (Rarbg) new limits/changes

[rarbg]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe

--

Describe the solution you'd like

No clue who's responsible for Prowlarr right now but @bakerboy448 said to open an issue so here is it

To fix Prowlarr potential issues please implement the following in your torrentapi code :

1. Increase the sleep between requests x2
2. if the API returns 429 retry in 2 minutes , 429 returns a custom json with rate_limit 1 - ignore it
3. if the API returns 520 retry in 3 minutes
4. if the API returns 200 with a json that contains rate_limit 1 retry in 5 minutes

That will cover all cases.

Describe alternatives you've considered

--

Anything else?

--

AB#3969

[Qstick]

Feel free to DM me on discord if you'd like to have a more in depth discussion on this. We can certainly change some things but it would be good to better understand things on your side.

[rarbg]

Please also change your app_id to rralworP to distinguish updated and non updated versions

[bakerboy448]

for posterity - context

when rated limited cloudflare will return http code 429 with a json encoded string that includes rate limit 1
when rated limited web returns http code 444 to cloudflare which returns 520 on cloudflare , there is no way to send 429 to cloudflare without it breaking the request properly and retrying it
when rated limited code returns http code 200 with rate_limit 1

rate_limit : 1 is generally returned when you've exceeded the rate limit or when one of the many servers behind cloudflare is being overloaded - because cloudflare doesn't evenly redistribute requests equally its there so clients via api can see that they are rate limited , retry in lets say 5 seconds and they can get their request fullfilled on another backend ( cloudflare doesn't do stickiness ) in x time

Jackett/Jackett#13663 (comment)

[Qstick]

Please also change your app_id to rralworP to distinguish updated and non updated versions

Why ban the old app_id, there was rate limiting in them as well? Prowlarr is not the problem, 90% of any volume coming from Prowlarr is just proxied from another app (Sonarr, Radarr, Lidarr, Readarr, etc. ). If you guys want to come up with a reasonable way to reduce volume, the apps, which are the source of the requests, are the way to do it. Again I invite you to come have an actual conversation about this, the amount of handling and hack fixes we have in place now to support this is getting comical.

[ilike2burnthing]

I don't think it's being banned, just make monitoring easier. Jackett did the same 4 months ago per request - Jackett/Jackett@e42c149

[Qstick]

All old versions with app_id Prowlarr are being reported as no longer working. Sounds kinda ban-ny to me.

I've been able to reproduce that Prowlarr doesn't work but even prowlarr does

[bakerboy448]

Confirmed Prowlarr is always 520'd by CF

2023-02-23 12:34:56.5|Trace|IndexerHttpClient|Res: [GET] https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=Prowlarr_10.0.0.22465: 520.520 (7252 bytes)

vs.

2023-02-23 12:31:34.4|Trace|IndexerHttpClient|Res: [GET] https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=rralworP_10.0.0.22465: 200.OK (77716 bytes) (1969 ms)

only diff was Prowlarr vs rralworP

[rarbg]

We are not doing this.

curl "https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=Prowlarr" -vvv 2>&1|grep 520
< HTTP/2 520

curl "https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=Prowlarr_version" -vvv 2>&1|grep 520
< HTTP/2 520

curl "https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=rralworP_version" -vvv 2>&1|grep 520
< HTTP/2 520

[bakerboy448]

curl I found always returned a 520 error no matter the app_id FWIW

testing with the dev build indicates the case appears to be otherwise as per the logs i shared

edit; seems intermittent actually?....so yeah Prowlarr doesn't seem blocked as of now, but certainly seems to get 520 more than anything else.

2023-02-23 12:44:08.8|Trace|IndexerHttpClient|Res: [GET] https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=Prowlarr_10.0.0.22465: 200.OK (77652 bytes) (3306 ms)

[rarbg]

@bakerboy448

curl "https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=Prowlarr" -A notcrl -vvv 2>&1|grep 520
< HTTP/2 520

curl "https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=Prowlarr_version" -A notcrl -vvv 2>&1|grep 520
< HTTP/2 520

curl "https://torrentapi.org/pubapi_v2.php?mode=search&ranked=0&limit=100&token=(removed)&format=json_extended&app_id=rralworP_version" -A notcrl -vvv 2>&1|grep 520
< HTTP/2 520

[rarbg]

Looks like its some weird CF caching bug
I flushed the cache completely
I don't see Prowlarr actually implementing any of the required timeouts from 3 months ago - WTF. Its literally shit ton of non compliant requests
Anyways i did some test after flushing and waiting for load to drop
30 second requests with app_id Prowlarr_123 , Prowlarr and rralworP_123 -

Thu Feb 23 19:53:44 CET 2023
< HTTP/2 200
Thu Feb 23 19:54:15 CET 2023
< HTTP/2 200
Thu Feb 23 19:54:45 CET 2023
< HTTP/2 200
Thu Feb 23 19:55:16 CET 2023
< HTTP/2 200
Thu Feb 23 19:55:47 CET 2023
< HTTP/2 200
Thu Feb 23 19:56:17 CET 2023
< HTTP/2 200
Thu Feb 23 19:56:48 CET 2023
< HTTP/2 200
Thu Feb 23 19:57:19 CET 2023
< HTTP/2 200
Thu Feb 23 19:57:49 CET 2023
< HTTP/2 200
Thu Feb 23 19:58:20 CET 2023
< HTTP/2 200

[bakerboy448]

yup cloudflare cache seems like it was the likely issue then

app_id Prowlarr_123 , Prowlarr and rralworP_123 is not a valid Prowlarr ID nor one that would be sent....unless 123 is a generic placeholder?

Prowlarr respect's the responses; will edit momentarily with the code refs

[rarbg]

@bakerboy448 there is no check if the version is int or any type of check actually
so i was lazy and used 123 for version placeholder
right now after flushing the cache there is the same issue with sonarr
no clue whats going on :)

edit - looks like sonarr also fixed itself

[bakerboy448]

initial call

Prowlarr/src/NzbDrone.Core/Indexers/Definitions/Rarbg/Rarbg.cs

Lines 115 to 143 in caab337

	protected override async Task<IndexerQueryResult> FetchPage(IndexerRequest request, IParseIndexerResponse parser)
	{
	var response = await FetchIndexerResponse(request);
	CheckResponseByStatusCode(response);
	// try and recover from token errors
	var jsonResponse = new HttpResponse<RarbgResponse>(response.HttpResponse);
	if (jsonResponse.Resource.error_code.HasValue)
	{
	if (jsonResponse.Resource.error_code is 4 or 2)
	{
	_logger.Debug("Invalid or expired token, refreshing token from Rarbg");
	_tokenProvider.ExpireToken(Settings);
	var newToken = _tokenProvider.GetToken(Settings, RateLimit);
	var qs = HttpUtility.ParseQueryString(request.HttpRequest.Url.Query);
	qs.Set("token", newToken);
	request.HttpRequest.Url = request.Url.SetQuery(qs.GetQueryString());
	response = await FetchIndexerResponse(request);
	}
	else if (jsonResponse.Resource.error_code is 5)
	{
	_logger.Debug("Rarbg temp rate limit hit, retrying request");
	response = await FetchIndexerResponse(request);
	}
	}

http response check/logic

Prowlarr/src/NzbDrone.Core/Indexers/Definitions/Rarbg/Rarbg.cs

Lines 48 to 63 in caab337

	public static void CheckResponseByStatusCode(IndexerResponse response)
	{
	var responseCode = (int)response.HttpResponse.StatusCode;
	switch (responseCode)
	{
	case (int)HttpStatusCode.TooManyRequests:
	throw new TooManyRequestsException(response.HttpRequest, response.HttpResponse, TimeSpan.FromMinutes(2));
	case 520:
	throw new TooManyRequestsException(response.HttpRequest, response.HttpResponse, TimeSpan.FromMinutes(3));
	case (int)HttpStatusCode.OK:
	break;
	default:
	throw new IndexerException(response, "Indexer API call returned an unexpected StatusCode [{0}]", responseCode);
	}
	}

Response checking / error handling

Prowlarr/src/NzbDrone.Core/Indexers/Definitions/Rarbg/RarbgParser.cs

Lines 27 to 63 in caab337

	public IList<ReleaseInfo> ParseResponse(IndexerResponse indexerResponse)
	{
	var results = new List<ReleaseInfo>();
	Rarbg.CheckResponseByStatusCode(indexerResponse);
	var jsonResponse = new HttpResponse<RarbgResponse>(indexerResponse.HttpResponse);
	if (jsonResponse.Resource.error_code.HasValue)
	{
	if (jsonResponse.Resource.error_code is 20 or 8 or 9 or 10 or 5 or 13 or 14)
	{
	var reason = $"{jsonResponse.Resource.error} ({jsonResponse.Resource.error_code})";
	if (jsonResponse.Resource.error_code is 5 || (jsonResponse.Resource.rate_limit is 1 && jsonResponse.Resource.torrent_results == null))
	{
	throw new TooManyRequestsException(indexerResponse.HttpRequest, indexerResponse.HttpResponse, TimeSpan.FromMinutes(5));
	}
	else
	{
	_logger.Debug("No results or imdbid/tvdb not found. Reason: {0}", reason);
	}
	return results;
	}
	throw new IndexerException(indexerResponse, "Indexer API call returned error {0}: {1}", jsonResponse.Resource.error_code, jsonResponse.Resource.error);
	}
	if (jsonResponse.Resource.torrent_results == null)
	{
	if (jsonResponse.Resource.rate_limit == 1)
	{
	throw new TooManyRequestsException(indexerResponse.HttpRequest, indexerResponse.HttpResponse, TimeSpan.FromMinutes(5));
	}
	return results;

[rarbg]

https://www.reddit.com/r/prowlarr/comments/11b7qa2/unable_to_connect_to_indexer_rarbg/j9wex6m/

Can you tell your mod to stop lying ?

[ilike2burnthing]

@PearsonFlyer update your comment, see above - #1169 (comment).

[rarbg]

@PearsonFlyer 's edit

EDIT: I've been asked to "stop lying" by the rarbg team. Rarbg is currently up and working with Prowlarr, at this moment.

[ftc2]

dear @rarbg,

thank you for your indexer!

however, in practice, it seems like i can only do about one request per minute now.

for the love of god, can you please, please have mercy on us and give us poor users a bit more to work with here? <3