v0.7.1

This release contains several minor optimisations and bug fixes.

Torrent pieces have been moved to their own table. This change mainly affects users who had enabled dht_crawler.save_pieces, and should improve search performance for those users.

What's Changed

• Move torrent pieces to separate table by @mgdigital in #134
• Reduce processor queue concurrency and increase timeout by @mgdigital in #135
• Upgrade go to 1.22 by @mgdigital in #136
• Upgrade GitHub Actions by @mgdigital in #137
• Fix importer handling of unclosed connection by @mgdigital in #138
• Check for "\x00" by @mgdigital in #139

Full Changelog: v0.7.0...v0.7.1

v0.7.0

This release moves the message queue from Redis to the Postgres database, and removes the Redis dependency.

Upgrading

Upgrade is automatic, but you can remove the Redis instance from your installation as it will no longer be doing anything. The docs and docker compose examples have been updated to reflect this.

What's Changed

• Remove Redis dependency and move message queue to Postgres by @mgdigital in #131

Full Changelog: v0.6.2...v0.7.0

v0.6.2

What's Changed

• Support Torznab query by season and episode by @mgdigital in #130

Full Changelog: v0.6.1...v0.6.2

v0.6.1

What's Changed

• Torznab imdbid optimisation by @mgdigital in #129: A simple optimisation that significantly improves Torznab performance when searching by IMDB ID.

Full Changelog: v0.6.0...v0.6.1

v0.6.0

This release introduces a significant optimisation of aggregations (counts).

It takes advantage of the fact that a Postgres query plan can tell you the cost of a query up-front, along with a rough estimate of the count based on indexes. All count queries now have a "budget", defaulting to 5,000. If the budget is exceeded according to the query plan, then the estimate will be returned (and the UI will display an estimate symbol ~ next to the associated count), otherwise the query will be executed and an exact count will be returned.

The accuracy of the estimate seems to be within 10-20% of the exact count in most cases - though accuracy depends on selected filter criteria and what is being counted, I've noticed bigger discrepancies but overall it seems like an acceptable trade-off.

The background cache warmer has been removed and aggregations are now real time again (the cache warmer was at best a short term mitigation while I figured out a better solution). The cache TTL has been reduced to 10 minutes. It was previously increased to allow the cache warmer to be run less frequently.

There are also some adjustments to the indexes that improve performance and the accuracy of estimations. For large indexes the migration may take a while to run - please allow it to run uninterrupted: in my tests on 12 million torrents it took 15 minutes.

What's Changed

• Use budgeted count for aggregations by @mgdigital in #128

Full Changelog: v0.5.1...v0.6.0

v0.5.1

What's Changed

• Fix issue with original language not being attached to content by @mgdigital in #123

Full Changelog: v0.5.0...v0.5.1

v0.5.0

v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked. See the updated docsite for more details on the new search syntax: https://bitmagnet.io/tutorials/text-search.html

Upgrading

The v0.5.0 upgrade process is fully automatic (unless you've used the /import endpoint, read below). All torrents must be reprocessed before they will work with text search. Just wait for the background job queue to work through them, then enjoy better and faster search!

If you have a large library the queue may take several hours, or even a day or more to work through. To check on progress, visit the /metrics endpoint and note metrics that contain process_torrent. The key ones being:

asynq_tasks_enqueued_total{queue="process_torrent",state="completed"}
asynq_tasks_enqueued_total{queue="process_torrent",state="pending"}
asynq_tasks_failed_total{queue="process_torrent"}

Until the queue has worked down you won't see any newly crawled torrents (though the crawler will be running!)

When you start the container for the first time you should see a progress bar for queueing the reprocess - please don't interrupt the process until you see the completed progress bar. If the process is interrupted you may have to requeue manually using the command bitmagnet reprocess --classifyMode=skip.

Note: I'm aware there are still some performance issues that need addressing for aggregations. These will be addressed separately - this release focuses on text search.

If you've used the /import endpoint, you will lose any classification hints that were part of the import. That is, if torrents were assigned to particular content types, video resolutions etc, this information will be lost and the classifier will attempt to classify from scratch without these hints. To recover the hints, the original import data must be reimported - it was necessary to separate these hints from the final classification result as the 2 things were previously conflated, making it impossible to separate out the data as part of this upgrade.

IMPORTANT: the format for the /import endpoint has changed slightly, please check the updated documentation at https://bitmagnet.io/tutorials/import.html (note there is a slight change to the jq query in the final code snippet).

What's Changed

• Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in #91
• Lazy instantiation for services with side effects by @mgdigital in #98
• Search query string rework by @mgdigital in #96
• Cache tweaks by @mgdigital in #99
• Add reclassify command by @mgdigital in #100
• Disable Gorm easer plugin which sometimes causes zero results by @mgdigital in #101
• Bump follow-redirects from 1.15.2 to 1.15.4 in /webui by @dependabot in #102
• Replace problematic chars in tsvector by @mgdigital in #103
• Fix search ordering by @mgdigital in #104
• Fix index out of range by @mgdigital in #105
• Refactor torrent processing by @mgdigital in #106
• Add 0.5.0 upgrade hook to queue reprocessing of all torrents by @mgdigital in #107
• Add missing criteria for collections by @mgdigital in #110
• Video classifier fixes by @mgdigital in #111
• Upgrade dependencies by @mgdigital in #112
• Reprocess tweaks by @mgdigital in #116
• Torrent processor optimisations by @mgdigital in #119
• Torznab: disable total count; fix index out of range by @mgdigital in #120
• v0.5.0 documentation updates by @mgdigital in #108
• Reduce crawler persistence batch size by @mgdigital in #121

Full Changelog: v0.4.1...v0.5.0

v0.5.0-beta.2

Thanks for giving this beta a try and reporting any issues!

v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked; the final release will include some documentation updates for the website that will go into further detail.

Upgrading

To try the beta, set your image tag to ghcr.io/bitmagnet-io/bitmagnet:0.5.0-beta.2.

What's Changed

Beta 2 builds upon Beta 1 with the following tweaks and fixes:

• Add missing criteria for collections by @mgdigital in #110 (fixes a bug with filtering by genre)
• Video classifier fixes by @mgdigital in #111 (video-related attributes were being incorrectly assigned to unknown content types)
• Upgrade dependencies by @mgdigital in #112
• Reprocess tweaks by @mgdigital in #116: This is the most important change in Beta 2, it significantly reduces the time needed for reprocessing all torrents (should be roughly 2x faster), by not making API calls that aren't strictly necessary. For users who have not configured a personal TMDB API key, this change is pretty much essential as the reprocessing time will be bottlenecked by the rate limiting on this API.

If you've already installed Beta 1

You don't need to do anything, but if your queue is still working away and you'd like to take advantage of the faster reprocessing you'll need to follow these steps.
If you've already installed Beta 1, AND you're using the default TMDB API key, then it's highly recommended to follow these steps:

1. Upgrade to Beta 2
2. Clear your queue manually by flushing Redis (e.g. docker exec -it bitmagnet-redis redis-cli flushall)
3. Re-queue your torrents using the CLI command reprocess --classifyMode=skip (e.g. docker exec -it bitmagnet bitmagnet reprocess --classifyMode=skip)

Full Changelog: v0.5.0-beta.1...v0.5.0-beta.2

v0.5.0-beta.1

Thanks for giving this beta a try and reporting any issues!

v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked; the final release will include some documentation updates for the website that will go into further detail.

Upgrading

To try the beta, set your image tag to ghcr.io/bitmagnet-io/bitmagnet:0.5.0-beta.1.

The v0.5.0 upgrade process is fully automatic (unless you've used the /import endpoint, read below). All torrents must be reprocessed before they will work with text search. Just wait for the background job queue to work through them, then enjoy better and faster search!

The queue may take several hours to work down if you have a large library. To check on progress, visit the /metrics endpoint and note metrics that begin asynq_tasks_enqueued_total{queue="process_torrent". Until the queue has worked down you won't see any newly crawled torrents (though the crawler will be running!)

When you start the container for the first time you should see a progress bar for queueing the reprocess - please don't interrupt the process until you see the completed progress bar. If the process is interrupted you may have to requeue manually using the command bitmagnet reprocess.

Note: I'm aware there are still some performance issues that need addressing for aggregations. These will be addressed separately - this release focuses on text search.

If you've used the /import endpoint, you will lose any classification hints that were part of the import. That is, if torrents were assigned to particular content types, video resolutions etc, this information will be lost and the classifier will attempt to classify from scratch without these hints. To recover the hints, the original import data must be reimported - it was necessary to separate these hints from the final classification result as the 2 things were previously conflated, making it impossible to separate out the data as part of this upgrade. IMPORTANT: the format for the /import endpoint has changed slightly, and the documentation for this on a branch here during the beta phase: https://github.com/bitmagnet-io/bitmagnet/blob/0.5.0-docs/bitmagnet.io/tutorials/import.md (note there is a slight change to the jq query in the final code snippet).

What's Changed

• Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in #91
• Lazy instantiation for services with side effects by @mgdigital in #98
• Search query string rework by @mgdigital in #96
• Cache tweaks (increase TTL and cache warmer interval) by @mgdigital in #99
• Add reclassify command by @mgdigital in #100
• Disable Gorm easer plugin which sometimes causes zero results by @mgdigital in #101
• Bump follow-redirects from 1.15.2 to 1.15.4 in /webui by @dependabot in #102
• Fix search ordering by @mgdigital in #104
• Refactor torrent processing by @mgdigital in #106
• Add 0.5.0 upgrade hook to queue reprocessing of all torrents by @mgdigital in #107

Full Changelog: v0.4.1...v0.5.0-beta.1

v0.4.1

What's Changed

• Remove asynqmon by @mgdigital in #93

Remove asynqmon altogether: The new version hasn't been published properly, and the previous version doesn't support the Redis wrapper.

Full Changelog: v0.4.0...v0.4.1