Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(search/cache): use cache to remove duplicate queries #682

Open
wants to merge 1 commit into
base: regex-lookup
Choose a base branch
from

Conversation

ShanaryS
Copy link
Contributor

@ShanaryS ShanaryS commented May 11, 2024

cross-seed search will now never perform duplicate queries. After searchee creation, the torznab search is calculated for each and those with the same are grouped back to back. When searching, the IndexerCandidates from the last search is cached and if it applies to the current search will be used (the map will ALWAYS have a size of 1, indexers not searched by previous are searched and added to cache). This allows duplicate queries to only hit each indexer once (per cross-seed search) and to be assessed near each other. This is mainly useful if you have multiple variations of a torrent like: different groups, resolution, source, etc which are not duplicates but will have duplicate queries. From my testing, my queries were cut in half.

These grouped searchees are ordered by most files first, then if .torrent based. We prefer most files first as that reduces the chance of a MATCH_PARTIAL happening when another searchee could have been a full match. I also changed how dupes are defined. Rather than just name, I check within the grouped searchees and find ones with the same size and number of files, preferring torrent based.

A nice benefit of the implementation is that if the previous searchee needed to search an indexer than the next wasn't going to, the results are still used by the next. Meaning that whenever we search for something, all searchees that care about the query will update their decisions, effectively ignoring excludeRecentSearch and excludeOlder if we are making the search anyways. This won't be the case initially but will eventually be true once a excludeRecentSearch cycle completes.


Updated some defaults in the config. Set snatchTimeout to 30s and searchTimeout to 2m. I think this is much better than leaving it at infinite. I also changed back maxDataDepth to 2. With the new caching, there is no risk of duplicate queries. And while partial reduces the need to go above a value of 1, in the cases of multi season releases 2 will be useful.

@ShanaryS ShanaryS added feature New feature or request logging dependencies Pull requests that update a dependency file search Related to search enhancement enhances existing features labels May 11, 2024
@ShanaryS ShanaryS self-assigned this May 11, 2024
@ShanaryS ShanaryS force-pushed the inject-saved-torrents branch 2 times, most recently from 877e54f to daa1a2c Compare May 12, 2024 01:04
@ShanaryS ShanaryS changed the base branch from master to regex-lookup June 14, 2024 22:32
@ShanaryS ShanaryS force-pushed the regex-lookup branch 2 times, most recently from 8a19650 to 19d16ef Compare June 14, 2024 23:34
@ShanaryS ShanaryS force-pushed the regex-lookup branch 3 times, most recently from a3af96d to a9cf39e Compare June 16, 2024 01:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
config related to config file enhancement enhances existing features feature New feature or request search Related to search
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants