Image uploads fail for certain filenames, files are orphaned in assets dir

[JanRomero]

Short description of the issue

Files whose filenames contain ©, ®, ™ and potentially other characters produce an error after uploading and cannot be added to the page, because the filename will end up having uppercase letters.

Because the error happens after the image is successfully copied to the page’s assets directory, the file will be orphaned there.

Expected behavior

Filenames should be sanitized consistently and the upload should work.

Actual behavior

Basically, the filename sanitation involves conversion to lowercase, which leaves the © untouched to be converted to an uppercase (C) by iconv() later. The uppercase letter remains in the filename that goes into the page’s assets folder. The file then can’t be found by ProcessWire for further processing, because it’s looking for it using an all-lowercase filename. I believe the call that eventually threw when I tested it was to filemtime().

Optional: Screenshots/Links that demonstrate the issue

I’ve posted a more verbose thread about this in the forums, but it didin’t get much traction.

Optional: Suggestion for a possible fix

I suppose moving the iconv call above the “beautification” block inside Sanitizer->nameFilter() would work, but I’m not sure if it would have other unwanted side effects. I don’t really understand why the conversion to lowercase only happens under specific conditions. Maybe it should just go at the end of the whole function.

Steps to reproduce the issue

1. Rename an image file to something involving a the character ©
2. Upload the file to a ProcessWire installation running on a case sensitive file system
3. After the upload ProcessWire will tell you the image is invalid
4. The image will still be inside /site/assets/files/<pageid>/
5. The filename will have turned from “Image © JanRomero.jpg” to “image_C_janromero.jpg”, for example

Setup/Environment

• ProcessWire version: 3.0.148
• PHP version: 7.3

[ryancramerdesign]

@JanRomero I can't seem to duplicate this one. I uploaded a file hello_©_world.jpg but the resulting filename after upload, and on the file system, is hello_c_world.jpg. Also tried replacing the underscores with spaces, like in your example.

I'm wondering if this might be related to locale settings, as you mentioned iconv(). Maybe I could duplicate it if I had the same locale settings that you do. Do you know what locale settings you currently have?

[JanRomero]

Hm, this definitely happens on the two different shared Apache servers I’ve tried it on. Both have locale de_DE.utf8, but iconv turns © into (C) under en_US as well. Here is a screenshot of the Tracy trace that appears on upload:

Not sure if these are any help:

Thanks!

[Toutouwai]

@ryancramerdesign, try with ® and ™ characters. On my system the © is coverted to lowercase but those other characters are converted to uppercase. So a file named "©®™filename.jpg" produces this in /site/assets/files/[page id]/:

[ryancramerdesign]

@JanRomero @Toutouwai While the © wasn't reproducing it here, when I tried the filename that Toutouwai mentioned with the ® and ™, I was able to reproduce it then. My installation still didn't seem to be affected (I guess it's not case sensitive despite unix), but I could see on the file system that the R and TM were uppercase in the original filename: c_R_TMfilename.jpg. I have pushed a fix for this. Thanks