New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

file name too long - cache #1907

Open
zenjabba opened this Issue Dec 11, 2017 · 13 comments

Comments

Projects
None yet
9 participants
@zenjabba

zenjabba commented Dec 11, 2017

Unable to write cache chunk due to filename being too long

2017/12/11 07:05:37 ERROR : worker-1 <ur76lghmk0lgu8no3i3i9iphqn8d0ri3ad7rhs2mhdn74igln7od06pnglvrkj77313471lo18omdv6526tit9h46t6k6r9llocpj69lb15a838i41tlekoth970eqvju2ct6pnsjpsqrnj0bm4soahn34q68turb7udlirm9ql16j216d5684fa3udr1fcapervhrknhpc7dggt3qvo5t8ooihp656ulthofgd4l9l68psbi388elk7dq3unkhk>: failed caching chunk in storage 0: open /tmp/rclone-cache-streamer2/streamer2_cache/s5dri5l6lhg70tqicfopng89h4/pkk7v920jgq9jbo3v1lg6goo80/p2kfesuq32ru2q93qh1otf7j2k/pau1o64bl8uiertgbck2vcpmt0pte6vqvfs9t321l4maac6gm1rigf9s7v62pm17f5ds07ter3if5r64p7iubj4kqskb3l1oofac4que2lcn97ab5eor2uqpvsu235bl/ur76lghmk0lgu8no3i3i9iphqn8d0ri3ad7rhs2mhdn74igln7od06pnglvrkj77313471lo18omdv6526tit9h46t6k6r9llocpj69lb15a838i41tlekoth970eqvju2ct6pnsjpsqrnj0bm4soahn34q68turb7udlirm9ql16j216d5684fa3udr1fcapervhrknhpc7dggt3qvo5t8ooihp656ulthofgd4l9l68psbi388elk7dq3unkhk/0: file name too long

rclone v1.38-223-g7c972d37β

  • os/arch: linux/amd64
  • go version: go1.9.2
@ncw

This comment has been minimized.

Owner

ncw commented Dec 11, 2017

The chunk before the /0 is 256 bytes long exactly - most linux file systems only allow 255 byte segments.

I guess this is normally stored on drive which allows very large path names, so this hasn't been a problem before.

What do you think @remusb ?

@ncw ncw added the Remote: Cache label Dec 11, 2017

@zenjabba

This comment has been minimized.

zenjabba commented Dec 11, 2017

or maybe hash the file names to 128 bytes?

@remusb

This comment has been minimized.

Collaborator

remusb commented Dec 11, 2017

It's the first time seen in cache cause the files are now stored on disk too. This wouldn't work on crypt + local either.

Sadly, mapping filenames would add complexity to cache which I don't think would benefit it right now. I would like to have this fixed by fixing crypt to be allowed to be wrapped by cache and in that way fix this, performance and cosmetic issues caused.

@remusb remusb self-assigned this Dec 11, 2017

@cal2195

This comment has been minimized.

cal2195 commented Dec 13, 2017

By that last comment, do you mean allowing remote->crypt->cache to work well?

And if so, is there a way to encrypt the cached contents? (Is that what the password is for when setting up a cache remote?)

@remusb remusb added this to the Known Problem milestone Dec 19, 2017

@Covernel

This comment has been minimized.

Covernel commented Feb 4, 2018

will it be fixed?

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 4, 2018

The easiest way is to make crypt work behind cache which is something I intended to do anyway.

@mvia

This comment has been minimized.

mvia commented Feb 13, 2018

Is there any chance for a fix soon?

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 13, 2018

So this particular issue can't be fixed in a traditional way. It's a limitation of the OS and the fact that cache writes on the disk rather than directly to the cloud provider to provide persistency across rclone restarts.

There are multiple options to overcome it:

  1. don't wrap cache under crypt -> this is not recommended now as it can cause 403s. The work done here should help with the eventual fix to allow this order: #1825
  2. use shorter file names to accommodate for crypt expanding the file name
  3. maybe others like checking how you can tweak the OS to allow longer file names (long shot)

Note that 1 isn't really a fix to this. It's just a workaround.
Out of curiosity, what's the real length of the filename that can't be written? There was recently a doc change that says that file names shouldn't be longer than 143 characters with crypt: #2040

@Covernel

This comment has been minimized.

Covernel commented Feb 13, 2018

Noted with Thanks.
I think 2 also not really a fix too.

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 13, 2018

Yes, I do agree, one shouldn't need to rename their files but there's not much we can do about an OS limitation either.
If cache would trim the file names then the encrypted version wouldn't be decipherable anymore. If rclone does this from crypt then you'll have a lot of people asking who's renaming their files.

If you already have files with more than 143 then it won't work anyway.

@jusher00

This comment has been minimized.

jusher00 commented Mar 21, 2018

@remusb Do you have any timeline on when your proposed workaround, wrapping gdrive -> crypt -> cache, will be working without 403s? It's the only thing I'm still having trouble with, as none of the files with long names are useable behind cache :/

@gryphonmyers

This comment has been minimized.

gryphonmyers commented Jun 17, 2018

@remusb really appreciate your work on the rclone mount cache! I am also curious about a timeline for addressing this issue, thanks

@d2dyno

This comment has been minimized.

d2dyno commented Aug 21, 2018

Having some of these issues on my end too. Also experiencing this for some files/folders.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment