Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File move fails with eCryptFS if filename too long #644

Closed
Pheromir opened this issue Jun 19, 2022 · 4 comments
Closed

File move fails with eCryptFS if filename too long #644

Pheromir opened this issue Jun 19, 2022 · 4 comments
Labels

Comments

@Pheromir
Copy link
Contributor

Pheromir commented Jun 19, 2022

LRR Version and OS
LRR Latest Nightly using Docker (on Synology DSM)

Bug Details
Using eCryptFS (which is used by Synology for encryption) limits the filenames to a specific limit (143 bytes, so max 143 ascii chars or less if using unicode chars).
As some archives from specific sites tend to have long filenames, moving the file from the temp directory within the docker container to the final path, which is mostly somewhere mapped to the host fs, fails.

Matching Logs

---
args:
- https://exhentai.org/g/2xxxx5/5xxxx9
- ~
attempts: '5'
children: []
created: 2022-06-19T10:57:38.93057Z
delayed: 2022-06-19T10:57:38.93057Z
finished: 2022-06-19T10:57:45.46258Z
id: '2024'
notes: {}
parents: []
priority: '1'
queue: default
result:
  category: ~
  id: c8a9f0f8a8a5649fd1239b386b5de3903f075ff5
  message: The file couldn't be moved to your content folder!
  success: 0
  title: Well, here was a very long file name that I kinda prefer not to share publicly
    so i wrote this text to replace the very long file name
  url: exhentai.org/g/2xxxxx5/5xxxxxx9
retried: ~
retries: '0'
started: 2022-06-19T10:57:38.98318Z
state: finished
task: download_url
worker: '120'

Screenshots
Windows UI language is in german, sorry for that, but that screenshot here just proves that the filename is too long
grafik

Additional stuff
There is already a line that shortens the filename in LANraragi::Model::Upload line 185

$filename = substr $fn, 0, 254 - length($ext) - length(".upload");

I don't know how much effort it is, but maybe it could be a good solution to add a setting for the filename to the archive settings, where the user can specify an own length limit that suits their system?

Also the shortening has to be added to uploaded archives, perhaps in LANraragi::Controller::Upload line 27?

my $tempfile = $tempdir . '/' . $filename;

Not sure if this is the first location where the archive gets stored on the filesystem, if not the upload could fail earlier.

@Pheromir
Copy link
Contributor Author

143 bytes, so max 143 ascii chars or less if using unicode chars

afaik kanji/hiragana take up to 3 bytes per character, while there are also some rare characters that use 4 bytes.
So for filenames using kanji, the filename limit would be at 47 characters.

Maybe something like this would be useful to determine to correct byte length (and let the user specify the byte length instead of character length in the settings)

@Pheromir
Copy link
Contributor Author

Did some testing - hope its ok that I don't make a PR for this, as I didn't add a way to configure the limit in the settings area - and got it to work with this:

in LANraragi::Controller::Upload between line 26 and 27 inserted:

        my ( $fn, $path, $ext ) = fileparse( $filename, qr/\.[^.]*/ );
        use bytes;
        $fn = substr encode_utf8($fn), 0, 142 - length($ext) - length(".upload");
        $fn = decode_utf8($fn, Encode::FB_QUIET);
        $filename = $fn . $ext;
        no bytes;

and in LANraragi::Model::Upload replacing line 185+186 with:

    use bytes;
    $filename = substr encode_utf8($fn), 0, 142 - length($ext) - length(".upload");
    $filename = decode_utf8($filename, Encode::FB_QUIET);
    $filename = $filename . $ext;
    no bytes;

the 142 in the substr lines is my filename bytelimit, so that should be replaced with a configurable variable

The encode/decode_utf8 functions require use Encode qw( encode_utf8 decode_utf8 );

@Difegue
Copy link
Owner

Difegue commented Jun 24, 2022

I think adding a configurable byte limit for downloaded files is probably too deep to make a setting -- I'd be inclined to just modify the current shortening functions to reduce the limit to 142, which is still plenty in most cases.

@Difegue
Copy link
Owner

Difegue commented Jul 1, 2022

I've reworked the lines you've given a bit to avoid relying on Encode, and stashed it away behind a toggle in settings. (While 142 bytes is plenty for most cases I'm pretty sure I'd get complaints if I made it the default...)

Thanks for putting in the work on this! 🎉 It made it pretty easy for me.

Image

@Difegue Difegue closed this as completed Jul 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants