History
I've not fully healthy yet but wanted to share a more stable setup I've been running for a few days now: v0.9.3-hotfix.10
- the refresh logic has been reduced to fetch 100 items at a time instead of 1000 in order for it to work more reliably. The refresh failing at least once causes it to no longer work, causing it to not detect changes
- api retries now work properly. Previously it wasn't working and unrestricting links are detected as failing (and needing repair) even though it was a just network issue because the retry logic is closing the request body prematurely. No more unnecessary repairs.
- unplayable reason: it will now be explicitly stated in the logs why an item is marked as unplayable
-
libxml2-utilsadded to the docker image so that it works out of the box with the plex_update.sh script - additional regex patterns to more accurately identify TV show episodes for filtering
- enhanced torrent fixer logic to handle more complex repair scenarios and to track status more effectively
Covered the last batch of reports on edge cases in the latest release v0.9.3-hotfix.9
- space in titles causes plex_update script to not process it correctly, this is now fixed
- repair replaces 'in progress' repairs of the same torrent as well
- fixed log spamming when your account is expiring
- redundant logs
- nothing
Couple of misses on the last hotfix so I need to release v0.9.3-hotfix.8
Reminder again: FYI I will be deleting the releases folder containing the previous version binaries on Feb 22, 2024. This actually costs me about $10/mo that's why I switched to releases.
- API timeout default set back to 60s but unrestricts will now follow donwload timeout which is by default 10s
- healthcheck should work properly again, no more unhealthy containers
- as usual, repairs causing duplicates
- nothing
-
addl_playable_extensionsin the config: if you are having issues with certain files being moved to__unplayable__directory, then add the extension here if it's not supposed to
Last release before I work on renames again: v0.9.3-hotfix.7
FYI I will be deleting the releases folder containing the previous version binaries on Feb 22, 2024. This actually costs me about $10/mo that's why I switched to releases.
- long startup times
- nothing
- nothing
well here we go again v0.9.3-hotfix.6 but this time I'm pretty sure I've squashed the fucking refresh bug. it was an edge case with repairs and the race condition it introduces. i've added semaphore locks here and there and that caused some lockups on a condition you only get with repaired torrents.
i'm gonna focus on dmm for now... i'm really sick of looking at people's zurg logs...
- refresh!
- none
-
downloads_every_minsto configure how often downloads mount should be refreshed (default is 60 mins) this is only relevant if you've setenable_download_mountto true
Re v0.9.5-hotfix.5 Hey, just to explain the differences:
Hotfix4 isolated worker pool for refresh and getting info and repairs
Hotfix5 unifies all worker pools into 1
Thats the only thing i can think that is affecting users not getting new content
v0.9.3-hotfix.4 I was hoping we won't reach this high hotfix# but here we are
- hopefully refresh as i rework repairs again
-
expose_full_pathbecause it was causing ID issues
- none
After 18 hrs of zurg work, a couple of DMM downtimes in between, 1,5k line changes, we got ourselves v0.9.3-hotfix.3! i'm not ignoring your messages people, just wanted to send this out before the weekend is over. i'll attend to your messages tomorrow
- repairs, ofc, the main reason why we got this hotfix
- plex scanner being stuck! i had them but i no longer experience them so report if you still do
- speed ? try just setting up a new library and benchmark. i had mine scan 100 movies in under 10 mins
-
preferred_hostsin the config: causing more issues than it should -
use_download_cachein the config: also causing issues -
realdebrid_timeout_secsin the config: replaced, see below
- added easy way to share your logs (when reporting issues on Github)
-
api_timeout_secsin the config: API requests timeout, changed default from 60s to 4s (increase this if you're having API timeout issues) -
download_timeout_secsin the config: download requests timeout, default is set to 2s -
repair_every_minsin the config: repair is now triggered on demand by broken torrents or by schedule as set here. Default is 60 mins. -
expose_full_pathin the config: this helps plex or jellyfin or infuse identify content better. Might break *Arrs if enabled.
zurg v0.9.3-hotfix.2 release:
It fixes some connection issues caused by tcp keepalive connections and Real-Debrid throttling zurg users. I set a limit there to conform with Real-Debrid's imposed limits. Also proactive repair is fixed so when it fails to check for certain torrents, zurg won't consider the torrent uncached and attempt to repair it even if it was an api failure. The logging bit of repair is also more descriptive now so you don't assume what's happening. Debug level logging has also been set as default and will stay until zurg reaches 1.0.
A lot of zurg issues are being reported on all directions and it has been hard to track them. Different discord channels and DMs, reddit posts and chats, and also on Github - it has been challenging trying to correlate all of them and the speed on fixing these issues is affected. I hope it's not a big ask to file everything on Github https://github.com/debridmediamanager/zurg-testing/issues/new/choose
Thanks all!
zurg update: v0.9.3-hotfix.1
- fix repair issues
- fix issues on marking torrents as unplayable (seems to be a glitch on realdebrid api but this is now handled)
- fix routing issues with Infuse (please use
:9999/infuse/instead of:9999/dav/)
v0.9.2 was received tremendously by the community and I'm often seeing my work on places I didn't expect it to be. Today, after making sure all of the added functionalities work as it should be, I'm releasing v0.9.3! More than a month's worth of work has been poured into this and a lot of feedback from many, many Plex server admins have been incorporated. I promised renaming files/moving files/artificial sorting to come to this version but the refactor needed to pull it off might cause too much instability for this version. Hence, I decided to release now rather than later. Enough blabbering! Here are the changes:
Proactive cache status checking. Every torrent in your library is continuously checked if it is still cached in RD or not, enhancing availability for all your files. If you have "repair" functionality enabled, it means your torrents will always be playable as long you keep zurg running.
"Unplayable" directory. If you're annoyed by the many rar-related log messages, or you also use Real-Debrid to download other non-video content like games or books, then expect those torrents to be found on __unplayable__ (instead of movies) from now on. Just to clarify, "rar'ed" torrents are no longer hidden, they're now sent to this directory.
CDN (real-debrid.cloud) streaming fix. If you've experienced frequent "playback restarts" when using RD's CDN hosts, then this should now be fixed.
Fix only_show_the_biggest_file. I claimed in many occassions that this feature is working only to find out recently that I am using an unreleased version. It should now work as expected. Also, when deleting the only file shown in this torrent, it will also delete it in RD.
Add more file filters only_show_files_with_size_lte and only_show_files_with_size_gte. "lte" and "gte" means "less than or equal" (greater). In a certain directory, you can add >= number_of_bytes or <= number_of_bytes to get rid of the noise.
The on_library_update hook now works on Windows. It's using Powershell. Please send me your working Powershell scripts so I can add them in the repo.
Support for a network proxy has been added. In your config.yml you can add proxy: "socks5://[username:password@]host:port". HTTP, HTTPS and SOCKS5 proxies.
Downloads directory. All your unrestricted links can now be found on a new directory __downloads__. A lot of users requested this but honestly, I do not see the point. But hey, it's there now!
version.txt. You'll find a new file when you mount zurg that contains version information. Now, you can check if the mount is ready or not by checking for that file's existence.
New zurg utility commands: clear-downloads and clear-torrents. Only usable from the command line, it clears all your unrestricted links and your whole library in seconds (it doesn't send thousands of request to RD so you don't need to worry about rate limits).
Handle EOF errors, empty RD accounts. zurg now supports the neophytes!
Handle non-premium users and add warnings when your premium is expiring. zurg won't start anymore if it detects that you are not a premium user (instead of displaying confusing errors). zurg tells you 48 hours before expiry if you need to renew to prevent risk of losing your library. Backup often in DMM to be safe!
Username+Password protection. A lot of people are mindlessly exposing their zurg servers on public IPs so in order to prevent leaking stuffs, you can add username: and password: on your config.yml.
No need to delete data folder when changing values of retain_rd_torrent_name and retain_folder_name_extension. I also suggest the *Arr stack people to read the config.yml.example.
Network tests for ipv4, ipv6 and both. Run zurg network-test -h to check. I actually like this feature in particular because it load balances RD better.
Did I already mention that the repair functionality was reworked again for the Nth time? I recently found out that reinserting != repairing torrents in Real-Debrid. It works but only when another Real-Debrid user downloads that one broken file (e.g. an episode in a season pack) in RD which refreshes that particular file in RD's cache and so now when you reinsert the torrent, it picks up the newly cached file. Now, zurg does this all the time. Note that when zurg is repairing files, and you restart zurg, it loses track of the torrents it needs to delete and reinsert for repair.
Well... that's it! You've reached the end of the release notes. It's a long one this time but there's still more to come on future versions (renames and also a feature that will no longer need you to do renames). Happy 2024!
rdt-client users: I have found a fix to handle waiting_files_selection better! It should work now in v0.9.2-hotfix.4
Released v0.9.2-hotfix.3 to fix the repair issues once and for all!
Also added a way to specify log level with LOG_LEVEL env var e.g. LOG_LEVEL=WARN will only output warnings and errors, etc. It's INFO by default, set to LOG_LEVEL=DEBUG when debugging issues
v0.9.2-hotfix.2 is now available
- Fix the rar repair issue (if you have experienced torrents that are being auto-rar'ed by Real-Debrid no matter what files you select)
- Added
auto_delete_rar_torrentsto auto-delete these auto-rar'ed torrents (since there's no method to fix them as of yet) - Added
/logsendpoint and logs are also kept in files
I've released v0.9.2-hotfix.1
Docker image: docker pull ghcr.io/debridmediamanager/zurg-testing:v0.9.2-hotfix.1
Binary download: https://github.com/debridmediamanager/zurg-testing/tree/main/releases/v0.9.2-hotfix.1
- file listing update issue fixes
- file delete issue fixes
- added
size_gtesize_gteany_file_inside_size_gteandany_file_inside_size_ltefilters - debug endpoint now contains loaded config
- on library update hook shows the output again
Just try it and lmk
This is it! v0.9.2 is now available. It's a 2.7k line update which comes with breaking changes (read the instructions carefully in order to not lose your library). I just really wanted to move on from this memory leak issue so that we will have a peaceful binge watch for the holidays with the fam.
Enjoy!! :snowman2:
Step 1: stop Plex server Step 2: stop zurg rclone mount Step 3: download the new version
if you are using docker:
docker pull ghcr.io/debridmediamanager/zurg-testing:v0.9.2
docker compose downif you are using the binary:
wget https://github.com/debridmediamanager/zurg-testing/raw/main/releases/v0.9.2/zurg-v0.9.2-linux-amd64.zip
# or just get them from the repo
# and then remember to stop zurg!Step 4: delete the contents of your zurg's data/ directory
Step 5: if you are using webdav, change webdav rclone config (rclone.conf) and point it to http://localhost:9999/dav/
Step 6: start zurg
Step 7: Go to http://localhost:9999/ to confirm if zurg is running
Step 8: start rclone mount again
Step 9: Check if the mount is working first ls -1R /pathtoyourzurgmount
Step 10: If everything works, start Plex server again
So as of v0.9.2 ...
Here are the only relevant configs for zurg, the rest have been removed:
- host
- port
- concurrent_workers
- check_for_changes_every_secs
- enable_repair
- on_library_update
- retain_folder_name_extension
- retain_rd_torrent_name
- preferred_hosts
- network_buffer_size
- serve_from_rclone
- force_ipv6
- realdebrid_timeout_secs
- retries_until_failed
- use_download_cache
Yes use_download_cache is now disabled by default
Output logging has been remove from on_library_update hook
Deleting torrents also trigger on_library_update hook
Fixed the issue on deletes not working Also apparently the file deletes aren't retained properly across zurg restarts, it should be working now BUT you will see the file you have deleted previously unless that triggered the whole torrent being deleted (by deleting all the files inside)
The router has been completely reworked to handle more concurrent requests, that's why I had to change the dav route
Torrent manager is the god class in zurg and has been completely reworked
You can now see the version if you run zurg version
Or you can also pass the config file path now with flags, check zurg -h
A lot of performance tweaks here and there...
Please please please report issues here with all the details you can provide! https://github.com/debridmediamanager/zurg-testing/issues
v0.9.1-hotfix.2 Reliability hotfix also on :latest
As usual docker pull ghcr.io/debridmediamanager/zurg-testing:latest
- Added new configs (optional since there are sensible defaults set)
# reliability configs
unrestrict_workers: 10 # since unrestricting has a different rate limit, use a different worker pool. decrease this if you are getting 429s
release_unrestrict_after_ms: 100 # wait time for every unrestrict worker to be released. increase this if you are getting 429s
rate_limit_sleep_secs: 6 # wait time after getting a 429 from Real-Debrid API- Fix
__all__not loading properly, containers should not go unhealthy anymore - Fix memory leak issue and optimize general memory profile
- Fix Plex scanner being stuck issue (run this to check your system for stuck scanners
ps aux --sort=-time | grep "Plex Media Scanner")
if you pull the latest image (i just pushed a couple mins ago) i added these 2 configs to handle those unusual cases of realdebrid api acting up
realdebrid_timeout_secs: 60 # api timeout
retries_until_failed: 5 # api failures until considered failedthese are the default values when you dont set them
Anyone having issues on Real-debrid CDN hosts (ends with .cloud) should pull the new hotfix.2 update because the previous hotfix.2 images had issues with Cloudflare connections.
Just released v0.9.1-hotfix.1 (also latest) because I saw that Plex won't play error videos because the error video file size and original content file size is different and then it gets "stuck" for about a minute trying to read the faulty file (also rclone showing unexpected EOF errors because of the file size mismatch). It should no longer happen and Plex can scan expired files faster.
Also fixed repairs not triggering on unrestrictable links when I reworked the code there to fix the issue where zurg gets stuck when there are more expired torrents than concurrent_workers. It should now trigger properly.
No changes in configs necessary, just upgrade the image
v0.9.1 released! docker pull ghcr.io/debridmediamanager/zurg-testing:latest or :v0.9.1
Changelog:
- Fixed health check issue (if you still see zurg as unhealthy after upgrading, restart it)
- Before it throws you an error video, zurg's data fetcher will retry up to 5 times now
- zurg now runs on the main CPU thread to maximize performance
- Added
retain_rd_torrent_nameinconfig.yml, set totrueto make it truly compatible withrdt-client - Repairs should now prevent torrents from being renamed
- The config
on_library_updategets passed the updated directories now as arguments ` - Added
serve_from_rcloneinconfig.yml, set totrueif you want zurg to just pass the file URL to rclone for downloading - Added
force_ipv6inconfig.yml, set totrueif you want zurg to connect to Real-Debrid's IPv6 address - In progress torrents no longer gets listed in directories
- Adjusted the modification times displayed for torrent directories and files. Files will now show
endedproperty while directories showaddedproperty. - Timezone issue fixed for modification times
- The rclone
httpmount now displays modification times for files
Moving forward, unless a hotfix is needed for a critical bug in zurg, I am adjusting the release schedule to be every 2 weeks to give me more time to work on other projects. Enjoy!
🌈 IT'S BEEN A MONTH SINCE ZURG'S FIRST RELEASE! 🥳
You guys remember zurg-1000? 😅 That's exactly a month ago!
Today's special so here is a special release!
Binaries for all platforms
Check out the latest releases on https://github.com/debridmediamanager/zurg-testing/tree/main/releases/
Deletes now work on webdav
You can delete a torrent , and...
Deleting individual files also work
Just make sure you retain your data/ directory and delete info_cache_time_hours in your configs, it's not needed
If all files are deleted inside a torrent, the torrent is also deleted
Webdav now works as fast as http
Lots of optimizations were done and also, the key update:
Just add pacer_min_sleep = 0 in your rclone.conf
[zurg]
type = webdav
url = http://zurg:9999
vendor = other
pacer_min_sleep = 0
Note: be careful on editing rclone.conf as your mount might not work properly, best to stop Plex server first if you dont know what you are doing or your library might reset itself
Several new filters are added!
not_regex
any_file_inside_not_regex
any_file_inside_not_contains
any_file_inside_not_contains_strict
and possibly the best filter ever (can only be topped by anime: true)...
has_episodes: true
which is a detection algorithm of episodes found inside a torrent
my config.yml looks like this now
directories:
anime:
group_order: 10
group: media
filters:
- and:
- has_episodes: true
- any_file_inside_regex: /^\[/ # e.g. [SubsPlease]
- any_file_inside_not_regex: /s\d\de\d\d/i
shows:
group_order: 20
group: media
filters:
- has_episodes: true
movies:
group_order: 30
group: media
filters:
- regex: /.*/
Sponsors
THANK YOU!!! you know who you are Thanks to vanderruss for sponsoring me on github My patreon page is also live 🎇
Hey all! I had several fuckups on the releases recently. The latest tag was pointing to an unstable code 😨 for a long time and I think a lot of people tried it and obviously got a bad experience with setting up.
I've fixed this issue and did a ton of rework on v0-defuse to again resolve the very difficult programming problem of race conditions and hopefully the latest image should be at par if not better than the previous versions. ⛅
I've also complete reworked the repair functionality :tools: and it shouldn't recreate gazillions of torrents. The repairs happen on the hash level now (for maximum efficiency). If one file is broken inside a group of torrents with the same hash, it will check the whole directory for any other broken files and it will only redownload what is broken (it doesn't reinsert torrents like what others do because that is not 100% guaranteed to repair broken files). The health check has been removed because it was also causing issues for libraries that takes a long time to get complete information for.
FYI http is faster than webdav by a factor of 4.5 (i compare zurg by itself now, lol)
p.s
(I really need to start focusing on the fuse work or else zurg will never get to v1 aka climax)
I'm sleepy & tired but cannot forget to thank @godver3 @nometaisbeta @yeungdrizzy for being amazing project sponsors! Also first time I saw zurg being put on a guide written by @thebropuks which is just mind blowing for a month-old software to make that much impact. 🤯
Enjoy the rest of the weekend y'all!
New release! latest or v0-defuse doesn't mattery for now
I've reworked every part of the architecture
the repair function is fully rewritten, it should be behaving properly now :slight_smile: Added a proper http client with retry/unrestrict
this setting is now fixed retain_folder_name_extension , it is now working to make it more compatible with rdt-client
and I added preferred_hosts! As to how to determine what to put on preferred hosts list, check my script here https://github.com/debridmediamanager/real-debrid-network-test/blob/main/real-debrid-network-test.sh (you can put as many as you want)
I can finally work back again on the fuse branch. Time to sleep!
^ I've created a hotfix for this, please reconfigure your docker-compose files and use this image instead
docker pull ghcr.io/debridmediamanager/zurg-testing:v0-defuse
- It contains the logging improvements (no way to configure logging level yet)
- race condition fixes making it more stable
- the repair functionality apparently can't be disabled entirely so I've fixed it here as well
(no fuse mount yet, hence defuse because i made an effort to remove it as it's not yet ready)
Hi all, i've recovered and am back working on the project again :slight_smile:
Some updates before the weekend: i haven't reached the point of working on the renames and deletes yet, but the mounting works, and can see the directories and files. it is again magnitudes faster because no translation of data is happening between protocols as we don't need rclone as a middle layer anymore.
I've also improved the way logging works and log level should be configurable now.
There are also several race conditions reported by different users that I've resolved. Again the directory mapping for torrents should be more stable on the next release.
(the http and webdav ports were untouched so you could still use that as a "stable" fallback)
The problem I'm having now is the way bytes are read inside a file. If using http or webdav, large chunks are requested at a time. This is great because when the connection closes, zurg can immediately stop requesting the file from RD servers. No bytes wasted. If using a filesystem mount, the OS controls the read buffer and it seems to be just 1kb chunks? It's super inefficient so I'm currently working on my own implementation of a cache layer that divides the whole file into parts and parts are loaded as needed. This is the main blocker of an early release.
Anw, hope everyone had a nice weekend!
zurg progress update
eta on the rename/delete function is by end of this week it will only support linux for now, and you will no longer have to use rclone
i'm collecting non critical bugs that will also be included in the release
zurg will also be released as a binary apart from the docker image
Update for today:exclamation: I have fixed this issue on the latest release that's available to be pulled now
-
any_file_inside_regexand otherany_file_inside_*filters works now - added
network_buffer_sizein config.yml to let you configure the best buffer size that's most compatible with your internet speed. there's a perfect setting here that depends on MTU, connection speed etc. so I made this configurable. - the issues affecting playability of the files (the files being unplayable, etc.) should now be fixed
it should make things a lot more stable!
A new release should be available in the next 10 mins
- repaired files creating a different folder should not happen again (no need to delete anything here for this fix to work)
- if same filename and size, then it deduplicates the files inside a folder (shows you the latest file)
- directory mapping issue fixed, thanks @Deep-value !
I tried making an improvement on the responsiveness of the playback. If this is something you're interested on testing, pull the latest image :slight_smile:
This helps even if you're using filebot
New release
Add this on your config.yml if you want your Plex libraries to always be updated :slight_smile:
on_library_update: |
token="Your X-Plex-Token Here"
plex_url="http://plex.box"
sections_xml=$(curl -s "$plex_url/library/sections" -H "X-Plex-Token: $token")
section_ids=$(echo "$sections_xml" | grep -o 'key="[0-9]*"' | awk -F'"' '{print $2}')
for id in $section_ids; do
curl -s -X POST "$plex_url/library/sections/$id/refresh" -H "X-Plex-Token: $token"
done
echo "Plex library refresh triggered!"please pull the latest release that will be available in 10 minutes
docker image inspect ghcr.io/debridmediamanager/zurg-testing:latest | grep revision
should be 0fd2fa691e1c32aec9f7762274fc9941e9313925
auto repair is done :slight_smile:
https://github.com/debridmediamanager/zurg-testing new release available in a few mins
- directory mapping fix (hopefully no more weird issues!)
- new config doc https://github.com/debridmediamanager/zurg-testing/blob/main/config.md (optional) you might want to update your
config.ymlbased on what you see here in this doc
please test the latest release. unfortuntely i haven't created a proper docker release pipeline so every release uses the latest tag for now
another change again for rclone.conf https://github.com/debridmediamanager/zurg-testing/blob/main/rclone.conf no_slash = false
the new release includes repair functionality, tested on a few use cases
ok i think i have fixed the issues on the http mount mainly
- file metadata - file type and file sizes are now properly set (you need to update rclone config again and set
no_headtofalse - when updating the torrent list, all directories goes empty and then back to normal again (or sometimes the directory mapping just fails)
The docker image is building now and will be available in a few minutes
change from zurg-1000 to zurg-testing
it's built now, docker pull ghcr.io/debridmediamanager/zurg-testing:main
https://github.com/debridmediamanager/zurg-testing/blob/main/README.md
pushed a new release, i recommend everyone to pull that image and update your rclone configs
if you are using a dockerized rclone, the new config should look like this
[zurg]
type = http
url = http://zurg:9999/http
no_head = true
no_slash = true
i have now heavily tested this with the rclone usecase
benchmark based on https://github.com/debridmediamanager/zurg-testing/blob/main/experiment.md
it's around 5-6x faster
it seems davfs2 is crazy fast though based on @kakasanti 's tests :slight_smile: thank you!
@here https://github.com/debridmediamanager/zurg-1000
I think i will remove the stupid limit, it was supposed to send the message that this is a beta/test version so people will provide feedback, but i think it will make the test comparisons look silly