[BUG?] Download speed slow #296
Comments
|
I have the same issue and have always had that. My suspicion, based on nothing other than a hunch, is that real-debrid throttles the download speed when the API is used in stead of HTTP(S) download. It could also be a bug, indeed, but I have never had faster speeds on real-debrid with RDT |
|
I use Alldebrid Was using realdebrid and same,.no issue with JDL Something strange |
|
Are you sure that JDownloader with Real-Debrid uses the API? I use JDL with Real-Debrid and it doesn't ask for an API key, it asks for a username and password, which seems to indicate it uses HTTP(S) downloading. RDT requires an API key for Real-Debrid in stead of username/password, as opposed to JDL. Can't speak for Alldebrid, never used it I do get full speeds with JDL and Real-Debrid though, indeed. Same if I download through the browser |
So my mistake sorry. |
|
But I don't think it's use API for download but only for debris file and take the link no? |
|
Same issue on a 2Gig connection barely getting 100 Mbps (12Mbps) per download . Dev pls look into it , using internal downloader. |
|
I have same speed with aria2 Maybe need to use internal command like wget or axel 19mb max now on this screenshot, but have more all the day ( maybe less people ) |
|
i am almost sure that the speed issue is limited to docker then |
I don't think because when using wget on docker console, speed is good |
You are right. I have checked the RDT client interface during a download and somewhere in the download properties it mentions a HTTPS-link. So indeed, API only for link grabbing, which makes this even more weird. Good call
I have the same issue, or at least the same symptoms, with the Windows-native RDT client, as well as with a docker one. My first reply above to this issue was about Windows-native. I should have mentioned that |
Yes it's use this example |
|
Which version are you on ? |
|
Of RDT 2.0.37 but I don't try yet, was on 2.0.35 |
I'm also on 2.035 could it be related to this release ? i am unable to pull the new release , looks like we have to buld it |
I use docker-compose personally because watchtower don't find the image too |
|
did you try upgrading to 2.0.37 ? |
|
Yes this morning but I don't try to download new file yet, i'm not at home, can't test.now |
|
ok , let me know how it goes once you try, i tried pulling the latest image through docker-compose still ending up on 2.0.35 even though the tag has been updated
|
Do you delete the image first ? |
|
got it done had to use "rogerfar/rdtclient:2.0.37" instead of latest tag |
|
|
|
|
|
looks like ill have to go back to native windows app and see , will keep you posted |
|
Lemme know 😘 |
|
|
|
|
|
Ok need to check my docker-compose so |
|
i never got speeds above 20mBps on docker , the above screen shots are from Windows App
|
|
Will try to add network_mode: host when I'm go back home |
Yes i have 2 vcpu and 6GB on debian 12 virtual machine and running about 13 container and my cpu is I5 9400 with 6 core and 6 thread |
2 vCPU's for the entire system? No, I certainly don't think that is enough for a bunch of VM's and containers, especially not when you want to do heavy stuff like parallel downloads and especially not with 12 (!!) VM's running. Try this: First start a download in RDT downloader which will take a while to complete. Then, shut down all your other containers and all your VM's (or as much as you can, leave as little else running as possible) except for RDT downloader so that it's the only thing running. Also because you have 2 vCPU's and so likely only 4 threads, lower the "parallel connections" setting to 4 in stead of 8, one connection per CPU thread. Now, with nothing else running except for RDT downloader, see what speeds you can reach. If it's still not full speed, play around with the settings a bit more (especially parallel threads and cache, though you could also try out some chunk settings, who knows) while leaving everything else shut down. If you reach much higher speeds now, it's a resource issue Also, please specify what type of disk you are writing your downloads to. SATA SSD? NVMe SSD? SATA spinning disks? This is important, if you're writing to a spinning disk, that will be an issue if you want to download at high speeds. In RDT downloader in the last tab "speed tests", what disk speeds does it say you have if you run the test? Do this test after you have shut down all other containers and VM's, so that you get the full disk speed without something else using the disk This is my disk write speed for instance. I let RDT downloader write to an NVMe cache drive and then let sonarr and radarr move the downloads to slower storage If I let it download to spinning disks I would never reach full speed because those typically write at about 150MB/s so if there are other apps and services using that disk (especially when, like in your case, your docker host is a itself already a VM and you are running other VM's inside the host VM) that will lower that speed a lot. Spinning disks are awful to run host systems on, especially when you need high write speeds and there are many different things running and using that disk |
yea i think u are right, i'm gona put more ram and vcpu to vm and i use one HDD 7.2K RPM with Write speed 45.3 MB/s i try to shutdown all my container only 2 remain and change the parallel connections setting to 4 same thing happen download speed to stable.... |
Oh wow, that is waaaaaaaay too slow. You will never reach high download speeds with such a slow disk. It could be that the limited amount of vCPU's and RAM also play a part (I think so, because 45MB/s is way too slow even for a spinning disk, those should reach at least 100MB/s and usually higher) because the download is usually cached in RAM before being written to disk. This is almost certainly a resource issue, at the very least you need a faster disk but perhaps also more cores and RAM. Start with the disk though and see how much it iimproves if you get a faster one. SATA SSD should already be enough. Especially when you use that as a cache and let sonarr and radarr import and move the downloads to your slower disk after they have completed. Of course, if you have the money, an NVMe disk is much better still. Even if it's just a very small one (128GB or something) that you only use as download cache, you will see an enormous improvement (as long as your CPU and RAM can keep up with higher download speeds) If you don't have a lot of money (I know how that feels) just get a very small SATA SSD (128GB for instance, those are very cheap now) and use that as download cache and don't use it for anything else. That will do wonders |
yea i know... is my 16TB HDD disk for movie/series, I have a small name 128gb but i'm using esxi so i need to put in my server and affect to my vm and mappe the download directory to this drive ? |
|
You can install and run ESXi on a 16GB USB stick, just saying. ESXi doesn't read from or write to its boot disk at all once it has fully booted up, it runs fully in RAM. I have run ESXi from a USB stick for years without issue. So you could run ESXi on a USB stick (or get another small SSD, of course) and use your current 128GB SSD as a download cache. Fantastic solution if you don't have much to spend on new hardware. ESXi will not run slower from a USB drive because it doesn't use the disk at all, except for writing log files but that is negligible |
my esxi is installed on ssd of 128gb and a lot space remain i'm gonna try to add space remain on my vm and use it to cache download |
Ah, I had assumed that you were likely using the remaining space for your VM's. Okay, if you have space left that is indeed a good solution. And if that still doesn't give full speed, just backup your current ESXi configuration, install ESXi on a USB stick, boot it and then restore your current configuration to it. Then assign the SSD as a dedicated VM and download cache drive. You will want to avoid running anything from a spinning disk. VM's, containers, applications, services.... They will all slow down enormously when running on a spinning disk. They really need an SSD. Good luck! [edit] |
thank you very much i'm gonna try this, but you know if i can download movie to my ssd and after radarr and sonarr move it to my other 16TB HDD ? |
EDIT: ahaha that it very fast now |
|
It's possible, yes but it's a bit long to explain here. Sonarr and radarr will need access to both disks and they will need to know which path RDT downloader is using for its downloads. Both sonarr and radarr will need both the slow and fast disks mapped. And then you will likely have to use path mapping (inside the settings menu) for sonarr and radarr to tell them that the path /data that you have set up inside the RDT container as download directory, corresponds to (for example) /myslowdisk/data/downloads inside radarr and sonarr. Your setup will be different but you will likely need path mapping for this |
|
you are so good thank you my friend |
Wait, is that 128GB disk an NVMe disk? Because if not, that speed is impossible with a SATA SSD, so that would be an incorrect test result. This is only possible if it's an NVMe disk (or multiple physical SATA disks in a striped RAID array) |
You're welcome. Glad to help and glad it's running better now |
nop is the SATA SSD where my esxi is installed |
Then it's definitely an incorrect test result. 1.23GB/s is 100% sure impossible with SATA SSD. The SATA connection itself can reach a maximum of 600MB/s, so even if the SATA SSD could go faster, you will never get more than 600MB/s because the SATA port it's connected to can't go faster. The only thing that can reach such speeds are NVMe SSD's because they don't connect to SATA, they connect directly to PCI-Express, which can go much faster than 3GB/s even. The latest (PCI-Express version 5) NVMe disks can reach 7GB/s [edit] |
|
so how can be check the speed ? that definitely wrong to |
That test sometimes gives inaccurate results for me as well, though it's usually slower, not faster. What I usually do is test 4 to 5 times and see which result I get most of the time. If my results are: 2GB/s , 1.5GB/s , 2.2GB/s , 150MB/s , 2GB/s I assume that that 150MB/s result is incorrect and my real speed is between 1.5GB/s and 2.2GB/S |
same thing max i reach is 73mb/s ... |
Yeah, so 45MB/s to 70MB/s will be your real speed. If you don't have multiple SATA SSD's (at least two) in a striped RAID array and you don't have an NVMe SSD in stead of SATA, 1.23GB/s is physically impossible with SATA, there is no way to get that speed with a single SATA disk. 600MB/s is the absolute maximum you will reach with a single SATA SSD and it will likely be a bit slower in reality (500MB/s is realistic) |
no that download speed, write speed staying same 1.23 1.33 gb/s, and yea i have only 1 ssd |
|
by the way i'm changing all my stuff including cpu of my esxi for intel xeon E5-2698 v4 so i'm gonna wait to change all of this and retry all of this |
That sounds like a good plan. If you have the money for it, try to get an NVMe drive to run your VM's on. A small one is enough. If, for instance, a VM has three virtual disks, one boot disk and two data disks, just configure ESXi to put only the boot disk on the NVMe and the two data disks (as long as you don't need high speed on your data disks, RDT will need a higher speed disk for its downloads of course) on SATA SSD or even on just a spinning disk. The fact that your VM's boot disk runs on NVMe will have the single largest impact on the performance of the entire system, the difference between spinning disk and NVMe is gigantic and even between SATA SSD and NVMe the difference is still very large. It's more important even that your VM's are running on an NVMe (or SATA SSD if NVMe is outside of budget) than it is important that ESXi itself is running on NVMe/SATA. ESXi doesn't need a fast disk, VM's absolutely require it |
i have one nvme 980 PRO 2TB, now all my vm is on HDD 2TB u think i can copy all my vm to the nvme and register it ? |
Depends. If your current motherboard supports NVMe then yes. Just plug it into the motherboard NVMe slot, add the NVMe drive as a new datastore in ESXi (This will wipe it completely!!!) Then if you have vCenter, just right click on each VM in vCenter and choose "migrate" and then migrate only the storage. If you don't have vCenter, right click each VM in ESXi and choose "unregister" (don't delete!) all of them. They will disappear but don't worry, the files are still there they just aren't registered anymore. Then manually move al the VM files from your SATA datastore to your new NVMe datastore using the ESXi datastore browser and then just browse to the VM folders from within ESXi, right click the VMX file for each VM and choose "register" So example if you don't have vCenter. Unregister all your VM's Then move all their files to the new datastore using the ESXi file browser (or through commandline) right click each VMX file in each VM folder in their new locations and choose "register". Done |
okay seem pretty easy thank you i'm gonna do this EDIT: |
Hey Since I use your settings, I have this issue : Any idea ? [11:28:03 ERR] Download reported an error: Exception of type 'System.OutOfMemoryException' was thrown. Any idea how can I fix it ? Maybe I can reduce buffer size to 6000000? |
Your buffer size is almost certainly too large for the amount of RAM you have, yes. Lower it until the error goes away. I have 32GB of RAM available to that container so you can use that number along with my buffer size, compared to your RAM to deduce a buffer size that might work |
|
Yes I have 16Gb will try 50% of your size
Thanks
Le mer. 28 août 2024, 17:14, BlueBull010 ***@***.***> a
écrit :
… Here are my settings. This gives me 112MB/s download speed, all throughout
the download. I'm on the newest version (2.0.81) in a docker container
running on Unraid. Do note that this is backed by the download container
running on SATA SSD storage and writing the download to NVMe SSD storage.
CPU is a 16-core AMD 5950X with 4 cores solely dedicated to this container
and 32GB DDR4 RAM [image: image]
<https://private-user-images.githubusercontent.com/61655638/361419148-f443e4a9-dbdf-45cf-9311-108058b26673.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ4NTY3NTEsIm5iZiI6MTcyNDg1NjQ1MSwicGF0aCI6Ii82MTY1NTYzOC8zNjE0MTkxNDgtZjQ0M2U0YTktZGJkZi00NWNmLTkzMTEtMTA4MDU4YjI2NjczLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI4VDE0NDczMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRkMWM3NjA4ZDRkMmJmZWUyMzMwZDNkMDBmMTU4YzEzMmFmOGJlNzY5M2NmNTk1NWZiZDc3N2U3MDYyOGIwYjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.F9h8wOhuFARyAWFC86K6H7Zn-kDnF03jp5qkZa3DBV0>
Hey Since I use your settings, I have this issue : Any idea ?
[11:28:03 ERR] Download reported an error: Exception of type
'System.OutOfMemoryException' was thrown.
Any idea how can I fix it ?
Maybe I can reduce buffer size to 6000000?
Your buffer size is almost certainly too large for the amount of RAM you
have, yes. Lower it until the error goes away. I have 32GB of RAM available
to that container so you can use that number along with my buffer size,
compared to your RAM to deduce a buffer size that might work for your
amount of RAM
—
Reply to this email directly, view it on GitHub
<#296 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AH3BVGO25UVJI32RUJBA2YTZTXSNJAVCNFSM6AAAAAA33CMWZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJVGY2DGNRVHA>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
Hey, i done my upgrade do you have some discord to talk of something ? |
What version are you using?
2.0.32
What OS are you running?
Linux
Are you using Docker or as a service?
Docker
Hey, download speed is slow, if I use wget or axel, I have approx 90mb/sec, rdt have approx 4mb/sec max
Any idea ?
Thanks
The text was updated successfully, but these errors were encountered: