Fixed: (IPTorrents) use offset to set page field

[afpak]

Database Migration

NO

Description

The IPTorrents indexer doesn't currently handle offset searching and will hit the same page over and over again looking for more releases as a result:

log

2022-12-28 11:07:38.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:38.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 0, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:38.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:38.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 0, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:40.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:40.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 100, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:40.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:40.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 100, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:42.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:42.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 200, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:42.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:42.5|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 200, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:44.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:44.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 300, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:44.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:44.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 300, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:46.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:46.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 400, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:46.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:46.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 400, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:48.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:48.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 500, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:48.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:48.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 500, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:50.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:50.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 600, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:50.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:50.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 600, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:52.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:52.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 700, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:52.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:52.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 700, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:54.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:54.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 800, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:54.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:54.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 800, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:56.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:56.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 900, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:56.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:56.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 900, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)
2022-12-28 11:07:58.4|Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2022-12-28 11:07:58.4|Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] for Season / Episode:[], Offset: 1000, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070]
2022-12-28 11:07:58.4|Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=
2022-12-28 11:07:58.7|Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] for Season / Episode:[], Offset: 1000, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010, 5070] from 1 indexer(s)

This change reuses code from the NzbIndex indexer to set the p (page) field based on the offset value provided:

log

Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 0, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010]
Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?q=%2B(tt(removed))&73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=&p=1
Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 0, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010] from 1 indexer(s)
Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 100, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010]
Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?q=%2B(tt(removed))&73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=&p=2
Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 100, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010] from 1 indexer(s)
Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 200, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010]
Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?q=%2B(tt(removed))&73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=&p=3
Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 200, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010] from 1 indexer(s)
Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 300, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010]
Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?q=%2B(tt(removed))&73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=&p=4
Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
Debug|ReleaseSearchService|Total of 100 reports were found for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 300, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010] from 1 indexer(s)
Debug|Prowlarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
Info|ReleaseSearchService|Searching indexer(s): [IPTorrents] for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 400, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010]
Debug|IPTorrents|Downloading Feed https://iptorrents.com/t?q=%2B(tt(removed))&73=&26=&55=&78=&23=&24=&25=&66=&82=&65=&83=&79=&22=&5=&99=&4=&60=&p=5
Debug|ReleaseSearchService|Total of 55 reports were found for Term: [] | ID(s): IMDbId:[(removed)] for Season / Episode:[], Offset: 400, Limit: 100, Categories: [5000, 5030, 5040, 5020, 5010] from 1 indexer(s)

[Qstick]

What's the page size for IPTorrents? This has the possibility to result in odd paging if we don't also factor page size into the equation, especially if page size is variable.

[afpak]

IPT defaults to 50 results per page, users can increase it up to 255.

With this change users should receive the same or more results regardless of their IPT max results setting, at least with Sonarr. Users with their IPT max set to < 100 will still receive at most a single page of results, since Sonarr assumes there are no further pages when less results are returned than the 100 it asked for.

Users with their max set to 100 or greater will receive the same first page of results as before, plus additional pages rather than page 1 over and over.

It's not ideal and there is the possibility of mismatches as further pages are parsed. How do you handle this for other indexers? I'm new to the arrs but some quick research indicated it was basically on the user to set the site setting to 100 results for the best experience. It was suggested to Jackett that they warn the user about it however it doesn't look like that went ahead.

I suppose it could also be handled by Prowlarr fetching and parsing multiple results pages as necessary to ensure it returns the proper number of results to each Sonarr request though that seems a fair bit more complicated.

[mynameisbogdan]

@Qstick - FYI something similar to this PR I applied to Libble as well which works as intended, since you can deduce the page based on limit and offset.
In my case they say 50 items/page, but it's in fact dynamic due to grouped results, by a tracker similar to gazelle or an early version I think.

Prowlarr/src/NzbDrone.Core/Indexers/Definitions/Libble.cs

Lines 150 to 154 in 6e1bef1

	if (searchCriteria.Offset.HasValue && searchCriteria.Limit.HasValue && searchCriteria.Offset > 0 && searchCriteria.Limit > 0)
	{
	var page = (int)(searchCriteria.Offset / searchCriteria.Limit) + 1;
	parameters.Add("page", page.ToString());
	}

[Qstick]

Yea this works if we can control the page size to be equal to the limit that is passed in, but if user passes in Limit:50 and Offset:0, and the indexer page size is always 100 they get 50 from page 1 and then Limit:50, Offset:50 gets them 50 from page 2, skipping the 50 in between.

I agree, This logic works better than nothing, which is what we have now, but we should think about some smarter global paging logic (particularly for Cardigann) as a larger project.

This way we can handle correct paging for examples where If user wants 100 and indexer only returns 20 per page we should page to get 100 or vise versa if user wants 20 and indexer page size is 100 we take 20, and then take 20 more from that original page when user passes an offset, etc..