Encrypted ACD: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password? #802

Closed
Trentflix opened this Issue Oct 18, 2016 · 33 comments

Projects

None yet

8 participants

@Trentflix
Trentflix commented Oct 18, 2016 edited

I keep getting the below error message with Encrypted Amazon Cloud Drive - I don't get it with Encrpyted Google when doing the exact same thing (just trying to access files from the mount - sometimes remotely). It will show up 4 times and won't open the file:

File/Path/TEST.mkv: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?

Ubuntu 14.04 x64
"v1.33-69-gd8d1102β"
rclone mount EACD: /path/to/mount/ --allow-other &

@DurvalMenezes

This seems to be a duplicate of #677; Trent, can you please confirm?

@Trentflix

It's the same error message, the difference is they are getting it when uploading files. I'm getting it when trying to play video files via the mount. I used rclone copy via the command line to upload the files without issue or errors.

@aneuploid1

I am getting the same error on random videos. I am not even trying to play the videos. Plex is just scanning and bathing the metadata.

@ncw ncw added the bug label Oct 20, 2016
@ncw ncw added this to the v1.35 milestone Oct 20, 2016
@ncw ncw added the Remote: Crypt label Oct 20, 2016
@aneuploid1

I just updated my mac mini to 10.12.1 beta 5... reinstalled the latest rclone and I am having a LOT better performance. I did have one instance of the "failed to authenticate decrypted block - bad password?" but it seems to be working better for some reason.

@jkaberg
jkaberg commented Oct 24, 2016 edited

Seem's to happen a lot more often now (?), seem's like I get it on every file the scanner tries to scan

I'm using rclone v1.33-77-g93e8440β

@ncw
Owner
ncw commented Oct 24, 2016

Here is an experimental fix for that error. It will produce a different error instead of the spurious failed to authenticate decrypted block - something to do with the connection breaking would be my guess.

http://pub.rclone.org/v1.33-81-gef17f90-readfull-fix%CE%B2/

I'm on the trail of the underlying error, but I need to confirm I'm going in the right direction, so any testing appreciated.

@jkaberg
jkaberg commented Oct 24, 2016 edited

@ncw Thanks for looking into this, here's a log: +link censored+

I only tested as far as getting one failed, let me know if you need several

EDIT: I'd also just for the sake of it point out that while observing these error's I'm seeing 0 packet loss in ref to 'connection breaking'

@Trentflix

I'm getting the same error message as before. It only happens on my Encrypted ACD mount though, not my Encrypted google Drive mount.

@aneuploid1

I don't see a version for OSX

@ncw
Owner
ncw commented Oct 25, 2016

@jkaberg useful log thanks. I see you are using seeks which lead me into finding a data corruption problem with crypt+seek which I've now fixed. Have a go with http://beta.rclone.org/v1.33-81-g9d2dd2c/ (will be uploaded in 15-30 minutes) and tell me if that makes a difference.

@jkaberg
jkaberg commented Oct 25, 2016 edited

@ncw Unfortunately still the same error and just to be clear: the file in question play's fine from local disk (unencrypted ofcourse) :-)

+link censored+

EDIT: and for the sake of it, here it works somehow: +link censored+

@jkaberg
jkaberg commented Oct 25, 2016 edited

So after debuging this further it seems some files play just fine (just now I tested with 1h playback which worked without a hiccup for one file), but most won't (but local playback works every time). How can I debug this @ncw?

EDIT: scratch that. thoes files that play'ed fine get the error's when I started jumping forward (seeking)

EDIT2: when the error accours most time the playback will fail and rclone will stop fetching data, but once in a while it seems rclone will continue to fetch the file (while the player has stopt playing back any content/reciving data). perhaps a seperate bug?

@ncw
Owner
ncw commented Oct 25, 2016

After looking at your logs, I have an idea... Are the files which don't play all bigger than 10 GB? Can you try playing a file < 10GB (say 5GB) and compare with one > 10GB?

(10GB is the templink threshold which means these files are being fetched a different way - maybe that way doesn't support Range requests or something like that. I can't test myself as my Internet connection has gone to pieces :-( )

@jkaberg
jkaberg commented Oct 25, 2016 edited

I tried with a few, one at 6GB and another at 8GB which plays and seeks fine. And a few more around 14-15GB that errors out so you might be on to something here? :-)

I'll do more testing but it seems promising

EDIT: I'm postive it's the templink thing, 10GB+ files does not seek at all

@krog
krog commented Oct 25, 2016

I can confirm your theory. I can play and seek videos smaller <9GB (default threshold afaik). Anything larger is fetched with templink and always fails when seeking.

@jkaberg
jkaberg commented Oct 25, 2016 edited

Just for the fun of it I tried mounting with --acd-templink-threshold=20G but still same issue as here #204

Perhaps this helps (seeing as the backend is S3)? http://stackoverflow.com/questions/36436057/s3-how-to-do-a-partial-read-seek-without-downloading-the-complete-file

@ncw
Owner
ncw commented Oct 25, 2016

Great, looks like we are getting somewhere :-)

What I need now is someone to run rclone mount with -v and --dump-headers and try to play a bigger than 10 GB video and post those logs. Important Make sure you remove the Authorization: lines from the log with eg grep -v Authorization: <dirty-log >clean-log before posting.

@calisro
calisro commented Oct 25, 2016

I'm seeing this with smaller files when accessed via a nginx http server to the crypt file system. Seems related. Want a log with headers? I can open a new issue but it seems very related to me.

clean-log.gz

@jkaberg
jkaberg commented Oct 26, 2016 edited

And here is one from me: +link censored+

@ncw
Owner
ncw commented Oct 26, 2016

@calisro The errors I see in your log all look like read tcp 10.100.1.195:59112->54.165.17.201:443: use of closed network connection

Can you open a new issue with that log please, and also put a description of your setup with nginx - if I can easily reproduce the problem locally it will be much easier for me to fix.

@ncw
Owner
ncw commented Oct 26, 2016

@jkaberg I can see what is happening there is that rclone tries to download the file with a Range: header using the templink.

This gets redirected to download directly via S3 as it is bigger than 10GB, however rclone doesn't put the Range: header back on at that point so it downloads from the beginning which isn't what the crypto expects, hence the bad password error.

Luckily I've already written code to do that in rest.clientWithHeaderReset so it will be a question of gluing A onto B and it should work.

I'll post a beta in a bit for you to try...

@ncw
Owner
ncw commented Oct 26, 2016

Here is a beta with that fix: http://beta.rclone.org/v1.33-82-g5986953/ (will be uploaded in 15-30 mins)

@krog
krog commented Oct 26, 2016

Your fix looks perfect. With 1.33-82 I can play and seek in a file >10GB.

@calisro
calisro commented Oct 26, 2016

Also confirmed to work in my testing setup.

@jkaberg
jkaberg commented Oct 26, 2016 edited

Confirmed, working fine here :-)

@ncw
Owner
ncw commented Oct 26, 2016

Excellent :-)

@calisro
calisro commented Oct 26, 2016

@ncw No need to create a new issue. I have narrowed the issue that I am having down to unionfs and I believe it is similar to Issue #823.

@ncw
Owner
ncw commented Oct 26, 2016

@calisro would probably be a good idea to confirm that the bactrace you see from rclone mount is the same as the on in that ticket.

@calisro
calisro commented Oct 26, 2016

@ncw Yes. It is the same.

@ncw
Owner
ncw commented Oct 27, 2016

I found a bit of missing locking in the FUSE module which can also produce this error message. Here is a beta with the fix - let me know if it works: http://beta.rclone.org/v1.33-83-g8710741/ (will be uploaded in 15-30 mins)

@Coornail
Coornail commented Oct 27, 2016 edited

I'm still seeing this error with many processes accessing files in 83.
Now the error messages are coming in pairs:

2016/10/27 10:39:49 file1: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?
2016/10/27 10:39:49 file1: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?
2016/10/27 10:40:35 file2: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?
2016/10/27 10:40:35 file2: ReadFileHandle.Read error: failed to authenticate decrypted block - bad password?
...
@ncw
Owner
ncw commented Oct 27, 2016

@Coornail probably the best thing to do is to try to work out what is causing the issue in such a way I could reproduce it, then put that into a new issue, with a full log with -v from the latest beta.

Thanks

Nick

@ncw
Owner
ncw commented Nov 3, 2016

I'm going to close this one now as this issue is fixed.

@ncw ncw closed this Nov 3, 2016
@ncw ncw reopened this Nov 4, 2016
@ncw ncw closed this Nov 7, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment