Offset not working in API

[nonamethanks]

Have you checked our Troubleshooting page for your issue?

• I have checked the Troubleshooting page

Is there already an issue for your problem?

• I have checked older issues, open and closed

Have you read our Contributing Guidelines?

• I have read the Contributing Guidelines

Environment

Latest linuxserver/jackett docker instance (Jackett Version v0.20.2163).

Description

I was wondering why Sonarr couldn't find the latest One Piece episode from Jackett's AnimeBytes index, and I realized that the offset was not working and all pages were returning the same result. Trying with Nyaa I get the exact same issue. Here's a simplified example:

/api/v2.0/indexers/nyaasi/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=0&limit=1&q=One%20Piece
/api/v2.0/indexers/nyaasi/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=1&limit=1&q=One%20Piece
/api/v2.0/indexers/nyaasi/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=2&limit=1&q=One%20Piece
/api/v2.0/indexers/nyaasi/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=3&limit=1&q=One%20Piece

These return the exact same torrent. The results stay the same no matter the offset. If I set limit=100 I get the same 100 torrents no matter the offset. Via UI I'm able to find hundreds of different torrents instead. How come?

Logged Error Messages

No errors in the logs.

Screenshots

No response

[ilike2burnthing]

As far as I'm aware Offset is being used for pagination in Jackett. Only 4 indexers support pagination, and only 2 of those use query.Offset to do so (AnimeBytes and Nyaa aren't them).

With that said, Sonarr wouldn't use it even if it were available, it wants a maximum of 100 results per indexer, and just makes one request per keyword.

[nonamethanks]

With that said, Sonarr wouldn't use it even if it were available, it wants a maximum of 100 results per indexer, and just makes one request per keyword.

That doesn't sound right, if I look at the Sonarr logs it tried to fetch 1000 results.

2022-10-25 17:32:31.4|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=0&limit=100&q=One%20Piece+1037
2022-10-25 17:32:33.6|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=100&limit=100&q=One%20Piece+1037
2022-10-25 17:32:33.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=200&limit=100&q=One%20Piece+1037
2022-10-25 17:32:35.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=300&limit=100&q=One%20Piece+1037
2022-10-25 17:32:37.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=400&limit=100&q=One%20Piece+1037
2022-10-25 17:32:38.0|Debug|RTorrentProxy|Executing remote method: d.multicall2
2022-10-25 17:32:38.1|Debug|RTorrent|Retrieved metadata of 693 torrents in client
2022-10-25 17:32:38.2|Debug|Api|[GET] /api/v3/queue/details?seriesId=53: 200.OK (1 ms)
2022-10-25 17:32:38.3|Debug|Api|[GET] /api/v3/command: 200.OK (0 ms)
2022-10-25 17:32:38.4|Debug|Api|[GET] /api/v3/queue/details?seriesId=53: 200.OK (1 ms)
2022-10-25 17:32:38.4|Debug|Api|[GET] /api/v3/command: 200.OK (0 ms)
2022-10-25 17:32:39.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=500&limit=100&q=One%20Piece+1037
2022-10-25 17:32:41.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=600&limit=100&q=One%20Piece+1037
2022-10-25 17:32:43.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=700&limit=100&q=One%20Piece+1037
2022-10-25 17:32:45.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=800&limit=100&q=One%20Piece+1037
2022-10-25 17:32:47.7|Debug|Torznab|Downloading Feed http://jackett:9117/api/v2.0/indexers/animebytes/results/torznab/api?t=search&cat=5070&extended=1&apikey=(removed)&offset=900&limit=100&q=One%20Piece+1037

In the end it failed searching for that specific episode because Jackett was only returning the same 100 torrents at every query, while animebytes has far more results for One Piece.

[nonamethanks]

Sorry for the double post, but @ilike2burnthing in Sonarr/Sonarr#5136 (comment) from August one of the Sonarr developers said that 1000 results are in fact fetched, so this sounds like a Jackett problem.

[ilike2burnthing]

I don't have an AnimeBytes account, but I don't see why it would be any different than any other indexer. I can't repeat that behaviour with RARBG or Nyaa, with a standard series or anime, searching automatically or manually, searching a series, season, or episode, or even specifically One Piece S21E146 (1037) as above.

The OP in the linked issue states this is seemingly random, even down to specific episodes in a series, so I could waste my day trying and still fail apparently.

The response you linked to isn't a Sonarr dev, they have made 2 commits, one of which was just plain text. They also just appear to be explaining what the user is seeing based on the logs, not necessarily that it should be happening.

The two devs that do reply don't actually explain why offset is being used, nor why it's seemingly random in its application. I suggest asking on their alternate support channels mentioned (ideally their forum or subreddit, in case you need to link to the answer).

[bakerboy448]

With that said, Sonarr wouldn't use it even if it were available, it wants a maximum of 100 results per indexer, and just makes one request per keyword.

Incorrect. Sonarr will paginate up to 10 pages (1000 results). This is based on the number of results indicated in the response as per spec. There needs to be more than PageSize as reported by the tracker/indexer (default and common: 100) results returned. Test Ref

Newznab/Torznab spec is pagination, are you saying that jackett fails to implement torznab spec?

The offset is pagination. It grabs 0-100 results. Then it grabs 101-200. And so on. 1000 results total, 10 API hits.

Originally posted by @PearsonFlyer in Sonarr/Sonarr#5136 (comment)

is accurate.

If jackett's trackers / implementation does not support torznab offset, then their caps should be updated to indicate as such.
https://github.com/Sonarr/Sonarr/blob/6660e0c3f37fade91d39141b2830dba4e20a46e2/src/NzbDrone.Core.Test/IndexerTests/SeasonSearchFixture.cs#L74

It appears setting caps.PageSize to 0 would disable paging *Arr side

[ilike2burnthing]

As far as I can see PageSize has never been implemented as a cap and exists nowhere in our code. The only uses of LimitsDefault and LimitsMax are for BTN, BHD, and the MetaIndexer.

Relevant open issues - #250 and #975.

[bakerboy448]

so #250 is the root cause of this specific issue given *Arr needs to be told to not paginate. This can be bandaided with a response and default value of 0 I guess.

#975 would need to be implemented in parallel with a proper #250 fix. to fully resolve this then.