Corrupt files when copying

[jdenman03]

qBittorrent & operating system versions

MAC OS 12.3.1
qBittorrent 4.4.2 x64

What is the problem?

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

HOWEVER, if I copy the file and paste it to the same HDD, the file sizes get corrected and then I can copy the file to my external HDD or to my NAS without any problems.

I initially thought that it was a MAC OS problem, but I upgraded to 12.3.1 and the problem is still there.

Steps to reproduce

No response

Additional context

No response

Log(s) & preferences file(s)

No response

[glassez]

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue

The screenshots above are identical, no?

I initially thought that it was a MAC OS problem, but I upgraded to 12.3.1 and the problem is still there.

Are you seriously sure that this update solves all the problems that existed before?

[jdenman03]

My screenshots are NOT identical. Look at the total disk space. Before the copy shows 1.92 and after the copy shows 1.98.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

[glassez]

Before the copy shows 1.92 and after the copy shows 1.98.

But the sizes in bytes are the same...
Also I can't see what kind of copying you're talking about. Maybe I didn't notice something again, but both files are in the same folder, and the full file name is not visible.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

Sorry, your sentence sounds for me like "since the problem wasn't solved by macOS upgrade then it wasn't the problem with macOS".

[glassez]

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

Maybe some files aren't completely flushed to disk at the time you encountered this issue.

[jdenman03]

Before the copy shows 1.92 and after the copy shows 1.98.

But the sizes in bytes are the same... Also I can't see what kind of copying you're talking about. Maybe I didn't notice something again, but both files are in the same folder, and the full file name is not visible.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

Sorry, your sentence sounds for me like "since the problem wasn't solved by macOS upgrade then it wasn't the problem with macOS".

Byte sizes are correct yes but the total size on the disk is wrong initially. If I copy that file to an external HDD or a NAS, the file becomes corrupt and the movie won't play correctly or be heavily pixelated. If I proceed to copy the file instead to the SAME HDD, the size on the disk then matches up correctly to the byte sizes and then I can copy the files to an external HDD or a NAS without any issues and the file isn't corrupt and my md5 checksum is the same.

I was just stating that the problem wasn't resolved by upgrading to the newest macOS. I don't have another mac to try, so I can't tell. I guess I could downgrade the qBittorrent to see if the problem fixes.

[jdenman03]

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

Maybe some files aren't completely flushed to disk at the time you encountered this issue.

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

[glassez]

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

[jdenman03]

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

Yes I will try that out next and let you know. I have a file that should be finishing in the next 30 minutes and will try that.

[jdenman03]

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

So quitting qbittorrent didn't do anything. However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot. I have attached screenshots for reference. However another file just completed after the reboot and the problem is still there.

So currently the only way to rectify the issue right now is to either a) reboot the machine after downloads or b) copy the file to the HDD first and then copy to the NAS/external HDD.

[ghost]

somewhat duplicate of #16787

[jdenman03]

somewhat duplicate of #16787

I saw that one, but mine isn’t 10x at all and while his is more, mine is less.

[ghost]

Do you use preallocate disk space?

[jdenman03]

Do you use preallocate disk space?

I edited my previous comment. But no I do not use preallocated space.

[glassez]

However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot.

I think it is yet another drawback of MMAP based disk IO subsystem of libtorrent 2.0. Most likely the buffers are not explicitly flushed to disk before closing the file, so the OS does not do it immediately.
@arvidn, ping.

[jdenman03]

However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot.

I think it is yet another drawback of MMAP based disk IO subsystem of libtorrent 2.0. Most likely the buffers are not explicitly flushed to disk before closing the file, so the OS does not do it immediately. @arvidn, ping.

This make sense. I am testing out a solution now to see if it fixes it. I will post an update after my next few downloads.

[jdenman03]

Also just a note and it looks like this has already been well discussed here on the forum, if I download directly to my NAS, qBittorrent will freeze and eventually hang up my entire mac causing me to hard reboot the machine.

[jdenman03]

My test solution didn't work. I tried using a separate path for incomplete torrents, but that didn't fix it.

[jdenman03]

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

[kheer1]

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

I'm having the same issue, what old build did you revert to? I'm thinking of doing the same thing.

[jdenman03]

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

I'm having the same issue, what old build did you revert to? I'm thinking of doing the same thing.

Just the old one that was already downloaded on my drive. It's 4.3.8

[arvidn]

I believe this is fundamentally caused by a bug in Mac OS. Writing to a memory mapped file is supposed to end up in the same page cache normal file I/O use as disk cache.

That said, the most recent libtorrent release attempts to hint the OS to be more aggressive in flushing data to disk, which may mitigate this issue.

[jdenman03]

I believe this is fundamentally caused by a bug in Mac OS. Writing to a memory mapped file is supposed to end up in the same page cache normal file I/O use as disk cache.

That said, the most recent libtorrent release attempts to hint the OS to be more aggressive in flushing data to disk, which may mitigate this issue.

while you are probably right and more advanced than me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

[arvidn]

while you are probably right and more advanced at me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

I don't believe it's using the new version of libtorrent. the new version of libtorrent uses memory mapped files for disk I/O

[glassez]

while you are probably right and more advanced at me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

I don't believe it's using the new version of libtorrent.

It uses libtorrent-1.2.

[ArthurGuez]

I'm having roughly the same issue. I download torrents directly onto my external hard drive but I have had a lot of corrupted files for the last 2 weeks or so. I'm not even able to install an older version of qBittorrent. I hope this will be fixed soon :)

[jdenman03]

I'm having roughly the same issue. I download torrents directly onto my external hard drive but I have had a lot of corrupted files for the last 2 weeks or so. I'm not even able to install an older version of qBittorrent. I hope this will be fixed soon :)

How are you not able to install an older version?

FYI, if you first copy the file to the same HDD it was downloaded on, it will flush the cache and it should be fine. That's what I resorted to until I installed 4.3.8

[arvidn]

@ArthurGuez what operating system are you using?

[thuglifem]

I dont have any problems with 4.4.5, as far as i remember I was on monterey 12.4 or 12.5 when it came out. Now I'm on ventura 13.0.1(mba m1). Works fine for me.

[arvidn]

@mbbananas you might want to check your RAM

[rainple-liu]

This problem should be caused by Apple. MacOS has been benchmark cheating in writes by not doing proper fsync.

[mbbananas]

Thanks! After months of trying different things, it seems that it is most likely the use of my USB hub. It was working before so it could be a combination of Apple and a USB hub. If I don't have too many externals connected to the same hub or I connect an external directly, everything seems to copy over fine.

[Stokestack]

Do we have a verdict on whether this is fixed in the current version, 4.5.2?

[loekf]

Yes, as long as you use the version with the older libtorrent.

[Stokestack]

Thanks. Was that version rolled back in the installer? Or is libtorrent installed independently? I don't remember installing it or where it resides.

[loekf]

There are two versions for each build. Just make sure you DONT download the version using lt20, because that one uses libtorrent 2.0 with a new file caching algo, which corrupts files on Macos.

[Stokestack]

Thanks. Why is that one still up, then?

A quick search didn't reveal any issues raised on this in the libtorrent repo, so I posted one.

[arvidn]

the problem people are talking about is still when copying the downloaded file, right? Not when playing/reading the file from the disk it was originally downloaded to. Is that right?

If so, I would be interested to know if it makes a difference to first stop the torrent and then copy the file, or maybe even shut down qbittorrent and then copy the file.

Also, just to be clear, the copying does not happen by qbittorrent itself, right? (e.g. move-download-location) but it happens via some other copying tool, like drag and drop in finder or cp or rsync.

[Stokestack]

That's right. The files seem to be fine when they reside where QBittorrent deposited them.

It doesn't make a difference if you stop the torrent. I've copied the files using Finder without qBittorrent even running, and they get corrupted. I tried two different drives (thumbdrive and SSD) over USB, and even copied the corrupted file back to the Mac... where it was still bad (showing that playback from the external drive wasn't the problem).

[MrGarak]

I have/had this exact issue as well with videos. I went crazy for months trying to sort it out, when it comes down to it, qBittorrent was causing this to happen for me. I would a video download to a folder locally on my mac mini, then copy to an external drive. Every single thing I copied would become corrupt and show these issues.

Switched to Transmission and have not had this issue once.

[michaelvolz]

I am using 4.5.5, and it seems I have now the same problem. Multiple files corrupted after moving to another drive. Is this a regression? I am the only one?

I moved the files back and made a recheck and indeed the files were incomplete, and some chunks needed to be downloaded again.

I enabled re-check after move and will see if it changes anything.

[thuglifem]

Which version of 4.5.5 are you using? The standard or the lt20? As far as I'm aware people don't have problems with the standard version, just with the lt20 one.
p.s. I don't have any issues with the standard 4.5.5, I can confirm that because it's not brand new version, i've been using it for a while.

[ram4life]

I've found that once a torrent finishes, right-clicking on it and doing a "force recheck" two or three times allows the client to check all the parts and I don't get the playback issues anymore. maybe this'll help?

[Stokestack]

Has the lib still not been fixed?

[michaelvolz]

fyi: My Issue is resolved, it was NOT QBittorrent.

After the move, the path was too long and produced a false file corrupted error from the Installer.

[datcactus]

I can confirm that even on a non-Mac system this is very easy to replicate with larger torrents (>10GiB in size). Downloading to an SSD and then moving (via NFS) to a ZFS array regularly causes 99.9% recheck failures which require moving back to the download directory to re-fetch the .1% corruption, move back to the array, and hope for the best. This is present on 4.6.2 so there has been no fix/improvement in this behavior over the 4.5.5 version I was on previously.

[rebusx]

@arvidn

the problem people are talking about is still when copying the downloaded file, right? Not when playing/reading the file from the disk it was originally downloaded to. Is that right?
Also, just to be clear, the copying does not happen by qbittorrent itself, right? (e.g. move-download-location) but it happens via some other copying tool, like drag and drop in finder or cp or rsync.

If you still need this info: in my case, i moved the files via "change download location" in qbittorrent like you say, during downloading. The torrent was huge, 200 GB or so, and individual files would be 1-3 GB. That is 100% sure.
Also relevant, another download like that was happening at the same time, also big one, and at least one of them gave me hash errors. I did a recheck on that one.
Ended up with an interesting corruption where a video file is mixed up with another file from that other download, i.e. a segment of another file had replaced a segment that is supposed to be. I'm certain this is not a filesystem or hardware issue, because none of the other large or media files on that disk got this problem, only the ones donwloaded with qbittorrent; and also, there was no any glitches in audio, which are guaranteed to be when there is an FS problem. In other words, a wrong segment was perfect by its data (video and audio tracks without issues), except it was from a wrong media file which of course gave structure errors in that file. So it looks exactly like "wrong piece".
Also, this was on NTFS in Windows, an external HDD connected via USB. But i can't tell you any versions :( this download happened in 2021 and i don't use qbittorrent for some time now

[eisenbruch]

I had this issue on my mac for the last few weeks it seems. Was driving me nuts.

I switched to using v4.6.3 NOT THE LT20 VERSION and it seems to have resolved the issue so far.

It seems insane that this bug still exists

[wpwentzell]

I can confirm that even on a non-Mac system this is very easy to replicate with larger torrents (>10GiB in size). Downloading to an SSD and then moving (via NFS) to a ZFS array regularly causes 99.9% recheck failures which require moving back to the download directory to re-fetch the .1% corruption, move back to the array, and hope for the best. This is present on 4.6.2 so there has been no fix/improvement in this behavior over the 4.5.5 version I was on previously.

+1. I stumbled onto this issue today after I had been backing up downloaded files to a local SMB share on another machine. Recently, I installed Plex on that machine and used those backup drives as media drives for Plex, and noticed that files larger than 10GB or so were glitchy/corrupted and would not play back correctly. After running a bit-by-bit comparison check using FreeFileSync, I noticed it was only the larger torrent files that had been corrupted. This is really frustrating, because now I cannot trust my files to get backed up without corruption. Not sure where the problem lies, but it's only happening to files that were downloaded using qbittorrent and then copied over to the SMB share.

I'm on Windows 11 and was using qbittorent 4.4.3. I have since updated to 4.6.3 to see if it alleviates the issue.

[vincejv]

+1
i can confirm this issue as well not mac, but on linux BTRFS ssd to BTRFS hdd moving files through incomplete/complete download feature on torrent client, it starts to seed after the files were moved from incomplete folder to complete then detects then midway it detects a "corruption", then redownloads the file again, probably due to the new broken disk io on v2, i'm reverting to Libtorrent v1.

Weird thing is it's consistent, when it's corrupted (doesn't happen all the time), only 1 piece (or the the last piece) gets corrupted and redownloaded, and the rest of the file seems to pass the hash check, so I don't see it being a hardware or ram corruption issue.

I have noticed this issue of Libtorrent 2 only, not on Libtorrent 1, using the latest Linux Kernel LTS v6.6.23. As previous users mentioned, mostly occurs when downloading very large files >20gb

@glassez Please consider this issue when QBT 5.0 is released, don't think we should rush on sticking with Libtorrent v2, i highly suggest to please keep both the libtorrent v1 and v2 builds.

[xavier2k6]

@vincejv @wpwentzell @eisenbruch How does qBittorrent 4.6.4 which uses Libtorrent 2.0.11.0 behave?

[vincejv]

if QBT 4.6.4 uses the unreleased 2.0.11.0, then the behavior is the same. I'm using QBT 4.6.4.

[glassez]

@glassez Please consider this issue when QBT 5.0 is released, don't think we should rush on sticking with Libtorrent v2, i highly suggest to please keep both the libtorrent v1 and v2 builds.

We returned to using libtorrent-1.2 by default a long time ago, when it became clear that the libtorrent-2.0 I/O subsystem was causing serious problems in some (fairly common) use cases.

[Stokestack]

@glassez Cool, thanks. So it's safe to let it auto-update?

[glassez]

@glassez Cool, thanks. So it's safe to let it auto-update?

If you are referring to the built-in updater, then it just downloads the default installer package for your OS, which is currently based on libtorrent-1.2.