Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrupt files when copying #16843

Open
jdenman03 opened this issue Apr 8, 2022 · 86 comments
Open

Corrupt files when copying #16843

jdenman03 opened this issue Apr 8, 2022 · 86 comments
Labels
Libtorrent OS: macOS Issues specific to macOS

Comments

@jdenman03
Copy link

jdenman03 commented Apr 8, 2022

qBittorrent & operating system versions

MAC OS 12.3.1
qBittorrent 4.4.2 x64

What is the problem?

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

HOWEVER, if I copy the file and paste it to the same HDD, the file sizes get corrected and then I can copy the file to my external HDD or to my NAS without any problems.

I initially thought that it was a MAC OS problem, but I upgraded to 12.3.1 and the problem is still there.

Screen Shot 2022-04-08 at 1 36 48 PM
Screen Shot 2022-04-08 at 1 38 17 PM

Steps to reproduce

No response

Additional context

No response

Log(s) & preferences file(s)

No response

@glassez
Copy link
Member

glassez commented Apr 8, 2022

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue

The screenshots above are identical, no?

I initially thought that it was a MAC OS problem, but I upgraded to 12.3.1 and the problem is still there.

Are you seriously sure that this update solves all the problems that existed before?

@jdenman03
Copy link
Author

My screenshots are NOT identical. Look at the total disk space. Before the copy shows 1.92 and after the copy shows 1.98.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

@glassez
Copy link
Member

glassez commented Apr 8, 2022

Before the copy shows 1.92 and after the copy shows 1.98.

But the sizes in bytes are the same...
Also I can't see what kind of copying you're talking about. Maybe I didn't notice something again, but both files are in the same folder, and the full file name is not visible.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

Sorry, your sentence sounds for me like "since the problem wasn't solved by macOS upgrade then it wasn't the problem with macOS".

@glassez glassez added the OS: macOS Issues specific to macOS label Apr 8, 2022
@glassez
Copy link
Member

glassez commented Apr 8, 2022

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

Maybe some files aren't completely flushed to disk at the time you encountered this issue.

@jdenman03
Copy link
Author

Before the copy shows 1.92 and after the copy shows 1.98.

But the sizes in bytes are the same... Also I can't see what kind of copying you're talking about. Maybe I didn't notice something again, but both files are in the same folder, and the full file name is not visible.

And no it does NOT solve the problem at all. I concluded the sentence out by saying "and the problem is still there."

Sorry, your sentence sounds for me like "since the problem wasn't solved by macOS upgrade then it wasn't the problem with macOS".

Byte sizes are correct yes but the total size on the disk is wrong initially. If I copy that file to an external HDD or a NAS, the file becomes corrupt and the movie won't play correctly or be heavily pixelated. If I proceed to copy the file instead to the SAME HDD, the size on the disk then matches up correctly to the byte sizes and then I can copy the files to an external HDD or a NAS without any issues and the file isn't corrupt and my md5 checksum is the same.

I was just stating that the problem wasn't resolved by upgrading to the newest macOS. I don't have another mac to try, so I can't tell. I guess I could downgrade the qBittorrent to see if the problem fixes.

@jdenman03
Copy link
Author

After downloading a file, there is a size discrepancy upon finishing. I have posted screenshots of the issue. What's odd is that the file will play perfectly fine on the same HDD it is downloaded to, but once I copy the file to an external drive or to my NAS the file becomes corrupt.

Maybe some files aren't completely flushed to disk at the time you encountered this issue.

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).
58052A20-D735-4F8E-BDF4-AC54CFCBF030_1_102_o

@glassez
Copy link
Member

glassez commented Apr 8, 2022

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

@jdenman03
Copy link
Author

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

Yes I will try that out next and let you know. I have a file that should be finishing in the next 30 minutes and will try that.

@jdenman03
Copy link
Author

It's every single download that is larger than 1GB. Here's what the videos do when I copy them to a external HDD or NAS (without copying them first to the initial HDD).

Could you test how it behaves if you close qBittorrent after downloading of some affected file is done? Also how it behaves if you reboot your Mac after downloading is done and before copying to the external HDD or NAS?

So quitting qbittorrent didn't do anything. However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot. I have attached screenshots for reference. However another file just completed after the reboot and the problem is still there.

So currently the only way to rectify the issue right now is to either a) reboot the machine after downloads or b) copy the file to the HDD first and then copy to the NAS/external HDD.
Screen Shot 2022-04-08 at 3 28 02 PM
Screen Shot 2022-04-08 at 3 30 57 PM

@ghost
Copy link

ghost commented Apr 9, 2022

somewhat duplicate of #16787

@jdenman03
Copy link
Author

jdenman03 commented Apr 9, 2022

somewhat duplicate of #16787

I saw that one, but mine isn’t 10x at all and while his is more, mine is less.

@ghost
Copy link

ghost commented Apr 9, 2022

Do you use preallocate disk space?

@jdenman03
Copy link
Author

Do you use preallocate disk space?

I edited my previous comment. But no I do not use preallocated space.
Screen Shot 2022-04-08 at 11 50 38 PM

@glassez
Copy link
Member

glassez commented Apr 9, 2022

However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot.

I think it is yet another drawback of MMAP based disk IO subsystem of libtorrent 2.0. Most likely the buffers are not explicitly flushed to disk before closing the file, so the OS does not do it immediately.
@arvidn, ping.

@jdenman03
Copy link
Author

However after rebooting the machine (without copying to internal HDD first), the file size corrected itself after the reboot.

I think it is yet another drawback of MMAP based disk IO subsystem of libtorrent 2.0. Most likely the buffers are not explicitly flushed to disk before closing the file, so the OS does not do it immediately. @arvidn, ping.

This make sense. I am testing out a solution now to see if it fixes it. I will post an update after my next few downloads.

@jdenman03
Copy link
Author

Also just a note and it looks like this has already been well discussed here on the forum, if I download directly to my NAS, qBittorrent will freeze and eventually hang up my entire mac causing me to hard reboot the machine.

@jdenman03
Copy link
Author

My test solution didn't work. I tried using a separate path for incomplete torrents, but that didn't fix it.

@jdenman03
Copy link
Author

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

@kheer1
Copy link

kheer1 commented Apr 20, 2022

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

I'm having the same issue, what old build did you revert to? I'm thinking of doing the same thing.

@jdenman03
Copy link
Author

just as an update to this post. I reverted back to a really old qbit build and everything works fine. I'll update back when there is a newer build and I will try it.

I'm having the same issue, what old build did you revert to? I'm thinking of doing the same thing.

Just the old one that was already downloaded on my drive. It's 4.3.8

@arvidn
Copy link
Contributor

arvidn commented Apr 20, 2022

I believe this is fundamentally caused by a bug in Mac OS. Writing to a memory mapped file is supposed to end up in the same page cache normal file I/O use as disk cache.

That said, the most recent libtorrent release attempts to hint the OS to be more aggressive in flushing data to disk, which may mitigate this issue.

@jdenman03
Copy link
Author

jdenman03 commented Apr 20, 2022

I believe this is fundamentally caused by a bug in Mac OS. Writing to a memory mapped file is supposed to end up in the same page cache normal file I/O use as disk cache.

That said, the most recent libtorrent release attempts to hint the OS to be more aggressive in flushing data to disk, which may mitigate this issue.

while you are probably right and more advanced than me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

@arvidn
Copy link
Contributor

arvidn commented Apr 20, 2022

while you are probably right and more advanced at me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

I don't believe it's using the new version of libtorrent. the new version of libtorrent uses memory mapped files for disk I/O

@glassez
Copy link
Member

glassez commented Apr 21, 2022

while you are probably right and more advanced at me at this, why is that 4.3.8 doesn't have this issue and everything works correctly?

I don't believe it's using the new version of libtorrent.

It uses libtorrent-1.2.

@ArthurGuez
Copy link

I'm having roughly the same issue. I download torrents directly onto my external hard drive but I have had a lot of corrupted files for the last 2 weeks or so. I'm not even able to install an older version of qBittorrent. I hope this will be fixed soon :)

@jdenman03
Copy link
Author

I'm having roughly the same issue. I download torrents directly onto my external hard drive but I have had a lot of corrupted files for the last 2 weeks or so. I'm not even able to install an older version of qBittorrent. I hope this will be fixed soon :)

How are you not able to install an older version?

FYI, if you first copy the file to the same HDD it was downloaded on, it will flush the cache and it should be fine. That's what I resorted to until I installed 4.3.8

@arvidn
Copy link
Contributor

arvidn commented Apr 27, 2022

@ArthurGuez what operating system are you using?

@ghost
Copy link

ghost commented Aug 31, 2022

or you may just download 4.4.5 which uses RC_1_2 by default now.

@loekf
Copy link

loekf commented Aug 31, 2022

or you may just download 4.4.5 which uses RC_1_2 by default now.

Briefly tried it, not a solution, because I see "couldn't write to disk" errors. Was also with 4.4.4 RC1_2.

RC2.0 or RC1.2 doesn't matter.

I suspect qbittorrent doesn't request write access to removable drives in my case, though the privacy settings look OK.

@ghost
Copy link

ghost commented Aug 31, 2022

I believe “couldn’t write to disk error” was not mentioned anywhere by OP and is diverting the topic at hand. It’s most likely a different issue and you should file a new issue ticket with detailed info such as logs, hardware details, configs.

@loekf
Copy link

loekf commented Sep 1, 2022

Looks like 4.4.x and 4.3.9 have seperate files to store settings, so 4.4.5 was trying to access an old external disk, which I replaced.

I tried 4.4.5. Copied one file to my NAS. Looks OK, but have to see how it behaves for a longer period.

@Steve-Agnew-NZ
Copy link

Confirming that upgrading to 4.4.5 has fixed the problem I was having with files downloading OK but then being corrupted by Finder as they were being copied.

@Guille-M
Copy link

Guille-M commented Sep 4, 2022

Copying issue (to external disks) when having Qbit open has been fixed with update 4.4.5, MacOS 12.5.1, no confirmed issues after copying and opening files.

@ghost
Copy link

ghost commented Sep 8, 2022

Copying issue (to external disks) when having Qbit open has been fixed with update 4.4.5, MacOS 12.5.1, no confirmed issues after copying and opening files.

Nothing was "fixed". Just the default build was changed to libtorrent 1.2.17. The problem should still exist if you try the RC_2_0 build.

@superfreeman1989
Copy link

I'm having the same issue with corrupted files. They all happen at ~80% progress with the video files. And I downloaded directly to external hard drive without copying. Only discovered the issue after downloading 200gb of stuff. Kinda pissed

@jdenman03
Copy link
Author

Have you upgraded to 4.4.5?

@danieljgmaclean
Copy link

Just sharing my experience for the record.

I started observing this same issue with video files that were downloaded both with qBittorent and Transmission. I tried different drives but they all had issues. The only things the files had in common with the problem was that they were downloaded using torrent and they were being moved to another drive after download.

I discovered that using the Finder to copy files nearly always corrupted the files. While using the terminal and the mv command, the files always copied 100% perfectly i.e. no stuttering in playback.

To be clear, no other files I download via the browser or other mechanisms resulted in corrupted files. For this reason, I ruled out APFS and macOS in general to be the issue, whether that's a good enough reason, who knows...

All my drives (internal and external) are formatted with APFS. I'm using macOS Monterey 12.3.1.

To fix this problem, I ended up writing a script to use the mv command to shift the files after download rather than doing it manually via the finder. This avoids the issue and the files copy fine and should mean I can use any version of qBittorent.

I've just updated qBittorrent to 4.4.5 and can see it's using Libtorrent 1.2.17.0.

If I see a repeat of the issue above, i'll post an update here.

@pifeifei
Copy link

pifeifei commented Oct 4, 2022

+1

Same question.

CPU: i3-3220
OS: TrueNas scale + docker

qBittorrent v4.4.5 Web UI (64-bit)

Qt: 6.3.1
Libtorrent: 2.0.7.0
Boost: 1.80.0
OpenSSL: 3.0.5
zlib: 1.2.12

@strpyw
Copy link

strpyw commented Oct 7, 2022

I went to qBittorrent to update my Mac OS 64 bit to v4.4.5
Did one download of 20G and copied the video file to one external drive.
Delete the torrent and reload it to the ext drive to check if there is corruption.
None, so far so good

@Oh4sh0
Copy link

Oh4sh0 commented Oct 12, 2022

Adding mine. Had qBittorrent 4.4.3 on macOS 12.4.

qBittorent would write to a local temp location and copy to a Synology NAS via SMB upon completion.

Upon completion, video files would have some sort of corruption where a few second segments would be missing throughout the videos.

Simply changing the download path to keep the files local seems to have solved the problem. I am upgrading to 4.4.5, and will revert my config to have qB copy direct upon completion and confirm whether the problem is resolved.

@mbbananas
Copy link

Has this worked for everyone? I've tried everything and this still just mysteriously stopped working earlier this year for me. The upgrade to qbittorrent 4.4.5 did not help. There is no work-around for me, the md5 hashes don't match most of the time and I get dropped frames and bleeding. Cp -r and rsync -r do not work. I've tried a lot of 3rd party file transfers like forklift as well to no avail. I even tried utorrent for a second instead of qbittorrent and got the same issue. I have several different externals and they all seem to be working fine. It's mostly on movie transfers and sometimes if I do it 5 different times, it will eventually work, but most of the time I get the dropped frames.

@thuglifem
Copy link

thuglifem commented Nov 11, 2022

I dont have any problems with 4.4.5, as far as i remember I was on monterey 12.4 or 12.5 when it came out. Now I'm on ventura 13.0.1(mba m1). Works fine for me.

@arvidn
Copy link
Contributor

arvidn commented Nov 11, 2022

@mbbananas you might want to check your RAM

@rainple-liu
Copy link

This problem should be caused by Apple. MacOS has been benchmark cheating in writes by not doing proper fsync.

@mbbananas
Copy link

Thanks! After months of trying different things, it seems that it is most likely the use of my USB hub. It was working before so it could be a combination of Apple and a USB hub. If I don't have too many externals connected to the same hub or I connect an external directly, everything seems to copy over fine.

@Stokestack
Copy link

Do we have a verdict on whether this is fixed in the current version, 4.5.2?

@loekf
Copy link

loekf commented Mar 28, 2023

Yes, as long as you use the version with the older libtorrent.

@Stokestack
Copy link

Thanks. Was that version rolled back in the installer? Or is libtorrent installed independently? I don't remember installing it or where it resides.

@loekf
Copy link

loekf commented Mar 28, 2023

There are two versions for each build. Just make sure you DONT download the version using lt20, because that one uses libtorrent 2.0 with a new file caching algo, which corrupts files on Macos.

@Stokestack
Copy link

Stokestack commented Mar 28, 2023

Thanks. Why is that one still up, then?

A quick search didn't reveal any issues raised on this in the libtorrent repo, so I posted one.

@arvidn
Copy link
Contributor

arvidn commented Mar 28, 2023

the problem people are talking about is still when copying the downloaded file, right? Not when playing/reading the file from the disk it was originally downloaded to. Is that right?

If so, I would be interested to know if it makes a difference to first stop the torrent and then copy the file, or maybe even shut down qbittorrent and then copy the file.

Also, just to be clear, the copying does not happen by qbittorrent itself, right? (e.g. move-download-location) but it happens via some other copying tool, like drag and drop in finder or cp or rsync.

@Stokestack
Copy link

Stokestack commented Mar 28, 2023

That's right. The files seem to be fine when they reside where QBittorrent deposited them.

It doesn't make a difference if you stop the torrent. I've copied the files using Finder without qBittorrent even running, and they get corrupted. I tried two different drives (thumbdrive and SSD) over USB, and even copied the corrupted file back to the Mac... where it was still bad (showing that playback from the external drive wasn't the problem).

@MrGarak
Copy link

MrGarak commented May 19, 2023

I have/had this exact issue as well with videos. I went crazy for months trying to sort it out, when it comes down to it, qBittorrent was causing this to happen for me. I would a video download to a folder locally on my mac mini, then copy to an external drive. Every single thing I copied would become corrupt and show these issues.

Switched to Transmission and have not had this issue once.

@michaelvolz
Copy link

I am using 4.5.5, and it seems I have now the same problem. Multiple files corrupted after moving to another drive. Is this a regression? I am the only one?

I moved the files back and made a recheck and indeed the files were incomplete, and some chunks needed to be downloaded again.

I enabled re-check after move and will see if it changes anything.

@thuglifem
Copy link

thuglifem commented Oct 9, 2023

Which version of 4.5.5 are you using? The standard or the lt20? As far as I'm aware people don't have problems with the standard version, just with the lt20 one.
p.s. I don't have any issues with the standard 4.5.5, I can confirm that because it's not brand new version, i've been using it for a while.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Libtorrent OS: macOS Issues specific to macOS
Projects
No open projects
Frequent/important issues
Core: file/torrent management and sto...
Development

No branches or pull requests