Problems with multibyte unicode characters in filenames #4513

georgehrke · 2013-08-19T10:27:38Z

Expected behaviour

ownCloud should handle multibyte unicode characters in filenames correctly.

Actual behaviour

ownCloud drops every character after (multibyte) unicode character in a filename

Steps to reproduce

Create a file like "%&:hankey:.pdf" on your computer
Upload it to ownCloud
After the upload, ownCloud displays the name correctly
reload the page
After reloading the page ownCloud shows only "%&"

Server configuration

Operating system:
Debian 7

Web server:
Apache

Database:
MySQL

PHP version:
PHP 5.4.x
The mbstring extension is enabled

ownCloud version:
latest master

Client configuration

Browser:
Chromium 30.0.1592.0

Operating system:
OS X

georgehrke · 2013-08-19T10:28:18Z

cc @icewind1991

DeepDiver1975 · 2013-08-21T21:27:06Z

@georgehrke Is the locale en_US.UTF-8 installed? THX

georgehrke · 2013-08-22T07:13:07Z

Yes and it's the only enabled in /etc/locale.gen

DeepDiver1975 · 2013-08-22T07:46:29Z

What unicode character did you use?
It's working on my Debian 7 with the violin symbol http://de.wikipedia.org/wiki/%F0%9D%84%9E

georgehrke · 2013-08-22T07:47:19Z

Pile of poo http://www.fileformat.info/info/unicode/char/1f4a9/index.htm

DeepDiver1975 · 2013-08-22T08:25:39Z

hmm - browsers on my debian are not capable of displaying the character properly:
http://en.wikipedia.org/wiki/%F0%9F%92%A9

Nautilus can't either.

I really wonder if we shall support all kinds of crazy characters out there.
No clue what will happen once we sync a file name with such a character to Windows

natschil · 2013-08-22T12:17:31Z

"I really wonder if we shall support all kinds of crazy characters out there."

Owncloud should definetly support UTF-8 in its entirety. It isn't up to owncloud to display the characters correctly though, that is the browser's responsibility.

karlitschek · 2013-08-26T09:17:00Z

That's indeed strange. ownCloud should fully support UTF-8. The fact that is shows up correctly after upload and is broken after reload indicates that the bug might be somewhere in the frontend/js code. Maybe a wrong encoding somewhere in an ajax call?

whitehairtong · 2014-03-25T06:11:08Z

I checked on the coding, and as far as I can see that in OC\LIB\PRIVATE\FILES\Mapper.php

the private function slugify($text) is seems doing a job to remove any unicode part of the file name and store only the ASCI part of the file name into physical_path.
private function slugify($text)
{
// replace non letter or digits or dots by -
$text = preg_replace('~[^\pL\d.]+~u', '-', $text);

// trim
$text = trim($text, '-');

// transliterate
if (function_exists('iconv')) {
    $text = iconv('utf-8', 'us-ascii//TRANSLIT//IGNORE', $text);
}

// lowercase
$text = strtolower($text);

// remove unwanted characters
$text = preg_replace('~[^-\w\.]+~', '', $text);

// trim ending dots (for security reasons and win compatibility)
$text = preg_replace('~\.+$~', '', $text);

if (empty($text)) {
    return uniqid();
}

When I do a test to change this function so that only $text is returned without any modification, the correct unicode part of the file path can be stored into mysql database for both logic_path and also physic_path in oc_file_map mysql databse 5.6.14 in windows environment.

without the modification, OC 6.0.2 will actually store files without the unicode part of the name into both the physic_path in mysql database and also the actual file name on the data folder

(unicode).pdf becomes -.pdf

Andrew

LukasReschke · 2014-03-25T08:16:34Z

Related: https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html

MorrisJobke · 2015-07-30T13:50:28Z

See #17978 (comment) for arguments for both ways.

jaller94 · 2016-09-26T11:19:13Z

This can still cause severe problems on ownCloud 9.0.5.2 with MariaDB 10:

All files with a 4 byte unicode character can't be edited, downloaded or deleted via ownCloud.
"Share" a White Screen of Death
Share your own root folder

Steps to reproduce

Preparation

Install ownCloud
Create user "Romeo" and "Juliet" (names don't matter)
Create a non-empty text file on your Desktop

"Share" a White Screen of Death

Name that file "love letter🐦.txt" (4 byte unicode somewhere in the name)
Upload it to Romeo's root folder. (Don't reload the page as it might break your ability to share the file.)
Share it with Juliet.

The result: After reloading the page Romeo sees this file as "love letter". From now on http calls from Juliet won't have any content (resulting in a white page). The server replies with 200 and doesn't log anything (no info, no warning, no error). Romeo is not affected by this white screen.

What happened?: The actual file in the file system got created correctly. The entry in the database does not contain the file extension.

Share your own root folder

Name that file "🐦love letter.txt" (4 byte unicode as the first character)
Upload it to Romeo's root folder. (Don't reload the page as it might break your ability to share the file.)
Share it with Juliet.

The result: After reloading the page Romeo sees this file as "files". Juliet sees a folder called "(2)". This folder contains all files from Romeo. Juliet also sees the file as "(2)" inside that folder "(2)".

What happened?: The actual file in the file system got created correctly. I think ownCloud stoped at the 4 byte character and shared Romeo's / instead of "/🐦love letter.txt".

My System

Server

Operating system: ubuntu-16.04.1-server-i386

Web server: Apache/2.4.18 (Ubuntu)

Database: 10.0.27-MariaDB-0ubuntu0.16.04.1

PHP version: 7.0.8-0ubuntu0.16.04.2

ownCloud version: 9.0.5.2 Community

Updated from an older ownCloud or fresh install: fresh install

Where did you install ownCloud from: owncloud.org

phpinfo: https://gist.github.com/jaller94/92fe27758d2394ba6cf2442d6e9578c0

Signing status (ownCloud 9.0 and above):

No errors have been found.

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'oc88qs1pfvqu',
  'passwordsalt' => '*** Removed for security reasons',
  'secret' => '*** Removed for security reasons',
  'trusted_domains' => 
  array (
    0 => 'localhost:8080',
  ),
  'datadirectory' => '/var/www/html/data',
  'overwrite.cli.url' => 'http://localhost:8080',
  'dbtype' => 'mysql',
  'version' => '9.0.5.2',
  'dbname' => 'owncloud',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'owncloud',
  'dbpassword' => '*** Removed for security reasons',
  'logtimezone' => 'UTC',
  'installed' => true,
);

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client

Browser: Firefox 45.2.0 and IE 11 on Windows 7

jaller94 · 2016-09-26T12:57:34Z

Uploading "..🐦.txt"

Uploading a file "..🐦.txt" into my home directory (bug using: trying to get access to the parent dir):

The result: I can't access my home directory anymore. An error is shown:
This directory is unavailable, please check the logs or contact the administrator.

owncloud.log:

Fatal   webdav  Exception: {"Message":"","Exception":"OCP\\Files\\InvalidPathException","Code":0,"Trace":"#0 \/var\/www\/html\/lib\/private\/files\/node\/node.php(179): OC\\Files\\Node\\Node->getFileInfo()\n#1 \/var\/www\/html\/apps\/dav\/lib\/connector\/sabre\/sharesplugin.php(162): OC\\Files\\Node\\Node->getId()\n#2 [internal function]: OCA\\DAV\\Connector\\Sabre\\SharesPlugin->handleGetProperties(Object(Sabre\\DAV\\PropFind), Object(OCA\\DAV\\Connector\\Sabre\\Directory))\n#3 \/var\/www\/html\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#4 \/var\/www\/html\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(999): Sabre\\Event\\EventEmitter->emit('propFind', Array)\n#5 \/var\/www\/html\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(919): Sabre\\DAV\\Server->getPropertiesByNode(Object(Sabre\\DAV\\PropFind), Object(OCA\\DAV\\Connector\\Sabre\\Directory))\n#6 \/var\/www\/html\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(334): Sabre\\DAV\\Server->getPropertiesForPath('', Array, 1)\n#7 [internal function]: Sabre\\DAV\\CorePlugin->httpPropFind(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#8 \/var\/www\/html\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#9 \/var\/www\/html\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(459): Sabre\\Event\\EventEmitter->emit('method:PROPFIND', Array)\n#10 \/var\/www\/html\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(248): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#11 \/var\/www\/html\/apps\/dav\/appinfo\/v1\/webdav.php(55): Sabre\\DAV\\Server->exec()\n#12 \/var\/www\/html\/remote.php(138): require_once('\/var\/www\/html\/a...')\n#13 {main}","File":"\/var\/www\/html\/lib\/private\/files\/node\/node.php","Line":78,"User":"jaller94"}   2016-09-26T12:45:56+00:00

Sharing it with another user

Before my home directory became unavailable I shared it with my Test-Account "tester" (because when I shared /🐦test.txt the user actually got / ):

The result: After the user "tester" logs into owncloud all http-requests get replied with 200 but don't contain any content. (white screen)

owncloud.log:

Error   PHP     TypeError: Argument 3 passed to OC\Files\Cache\Shared_Cache::__construct() must implement interface OCP\Files\Cache\ICacheEntry, boolean given, called in /var/www/html/apps/files_sharing/lib/sharedstorage.php on line 588 at /var/www/html/apps/files_sharing/lib/cache.php#63   2016-09-26T12:50:19+00:00

jaller94 · 2016-09-26T14:25:22Z

I was able to reproduce both behaviours in the current ownCloud 9.1.1.3.
The server setup is the same as in both of my posts above.

But instead of a White Screen of death Juliet gets a message when a file love letter🐦.txt was shared with her:

owncloud.log:

Fatal   cron    Argument 3 passed to OCA\Files_Sharing\Cache::__construct() must be an instance of OCP\Files\Cache\ICacheEntry, boolean given, called in /var/www/html/apps/files_sharing/lib/sharedstorage.php on line 325     2016-09-26T14:21:25+00:00
Error   index   Exception: {"Exception":"TypeError","Message":"Argument 3 passed to OCA\\Files_Sharing\\Cache::__construct() must implement interface OCP\\Files\\Cache\\ICacheEntry, boolean given, called in \/var\/www\/html\/apps\/files_sharing\/lib\/sharedstorage.php on line 325","Code":0,"Trace":"#0 \/var\/www\/html\/apps\/files_sharing\/lib\/sharedstorage.php(325): OCA\\Files_Sharing\\Cache->__construct(Object(OCA\\Files_Trashbin\\Storage), Object(OCA\\Files_Trashbin\\Storage), false)\n#1 \/var\/www\/html\/lib\/private\/Files\/Storage\/Wrapper\/Wrapper.php(387): OC\\Files\\Storage\\Shared->getCache('', Object(OCA\\Files_Trashbin\\Storage))\n#2 \/var\/www\/html\/lib\/private\/Files\/View.php(1362): OC\\Files\\Storage\\Wrapper\\Wrapper->getCache('')\n#3 \/var\/www\/html\/lib\/private\/Files\/Node\/Root.php(180): OC\\Files\\View->getFileInfo('\/Juliet')\n#4 \/var\/www\/html\/lib\/private\/Files\/Node\/Root.php(342): OC\\Files\\Node\\Root->get('\/Juliet')\n#5 \/var\/www\/html\/lib\/private\/Server.php(809): OC\\Files\\Node\\Root->getUserFolder('Juliet')\n#6 \/var\/www\/html\/lib\/private\/User\/Session.php(384): OC\\Server->getUserFolder('Juliet')\n#7 \/var\/www\/html\/lib\/private\/User\/Session.php(438): OC\\User\\Session->prepareUserLogin()\n#8 \/var\/www\/html\/lib\/private\/User\/Session.php(287): OC\\User\\Session->loginWithPassword(*** sensitive parameters replaced ***)\n#9 \/var\/www\/html\/core\/Controller\/LoginController.php(196): OC\\User\\Session->login(*** sensitive parameters replaced ***)\n#10 [internal function]: OC\\Core\\Controller\\LoginController->tryLogin(*** sensitive parameters replaced ***)\n#11 \/var\/www\/html\/lib\/private\/AppFramework\/Http\/Dispatcher.php(159): call_user_func_array(Array, Array)\n#12 \/var\/www\/html\/lib\/private\/AppFramework\/Http\/Dispatcher.php(89): OC\\AppFramework\\Http\\Dispatcher->executeController(Object(OC\\Core\\Controller\\LoginController), 'tryLogin')\n#13 \/var\/www\/html\/lib\/private\/AppFramework\/App.php(110): OC\\AppFramework\\Http\\Dispatcher->dispatch(Object(OC\\Core\\Controller\\LoginController), 'tryLogin')\n#14 \/var\/www\/html\/lib\/private\/AppFramework\/Routing\/RouteActionHandler.php(46): OC\\AppFramework\\App::main('LoginController', 'tryLogin', Object(OC\\AppFramework\\DependencyInjection\\DIContainer), Array)\n#15 [internal function]: OC\\AppFramework\\Routing\\RouteActionHandler->__invoke(Array)\n#16 \/var\/www\/html\/lib\/private\/Route\/Router.php(280): call_user_func(Object(OC\\AppFramework\\Routing\\RouteActionHandler), Array)\n#17 \/var\/www\/html\/lib\/base.php(891): OC\\Route\\Router->match('\/login')\n#18 \/var\/www\/html\/index.php(39): OC::handleRequest()\n#19 {main}","File":"\/var\/www\/html\/apps\/files_sharing\/lib\/Cache.php","Line":64}  2016-09-26T14:16:49+00:00

PVince81 · 2016-10-17T06:10:30Z

see #21899 for additional info

PVince81 · 2016-10-17T06:12:09Z

requires: adding tests for 4 byte unicode characters: Adding tests for 4 byte unicode characters #17978
requires: doctrine migrations: doctrine migrations #14851

@DeepDiver1975 anything else ?

PVince81 · 2016-12-12T09:02:51Z

@DeepDiver1975 doctrine migrations merged. Can you move forward with the 4 byte unicode PR ?

michaelstingl · 2016-12-20T14:32:35Z

for reference: 00006329

cdamken · 2017-03-07T19:41:23Z

Hi Guys, could you please tell me some feedback from this issue?

DeepDiver1975 · 2017-03-08T06:57:36Z

Fixed with oc10

PVince81 · 2017-03-08T08:42:01Z

raise doc ticket for configuration hints: https://github.com/owncloud/documentation/issues/2924

lock · 2019-08-02T20:00:37Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

ranek mentioned this issue Jan 31, 2014

Support characters in the Unicode Astral Plane #7030

Closed

karlitschek added Enhancement and removed Bug labels Jul 4, 2014

timw4mail mentioned this issue Aug 26, 2014

Cifs external storage : utf-8 problem ? #10625

Closed

bantu mentioned this issue Nov 4, 2014

Unicode unit test fail on local machine with mysql #11506

Closed

LeeThompson mentioned this issue Nov 11, 2014

ownCloud File Mapper Not Encoding UTF-8 Properly #12112

Closed

MorrisJobke removed the 2 - Triaging label Feb 27, 2015

DeepDiver1975 modified the milestone: backlog Mar 21, 2015

MorrisJobke mentioned this issue Jun 17, 2015

Certain UTF-8 characters break Mysql owncloud-archive/news#813

Closed

MorrisJobke mentioned this issue Jul 30, 2015

Adding tests for 4 byte unicode characters #17978

Merged

ghost mentioned this issue Aug 30, 2015

Import of tasks to database fails when there is an emoij in the file owncloud-archive/apps#2097

Closed

ghost mentioned this issue Dec 30, 2015

Multiple conflicts with emojis #21397

Closed

jaller94 mentioned this issue Sep 28, 2016

4 byte unicode characters in filenames with MariaDB causes severe bugs nextcloud/server#1560

Closed

PVince81 mentioned this issue Oct 17, 2016

Unicode Support #21899

Closed

PVince81 modified the milestones: 9.2, backlog Oct 17, 2016

PVince81 added green-ticket comp:core labels Oct 17, 2016

PVince81 mentioned this issue Oct 20, 2016

Creating tag putting an emoji makes dropdown unable to find tags. #22093

Closed

PVince81 mentioned this issue Nov 3, 2016

4 byte unicode in comments #24009

Closed

butonic added blue-ticket and removed green-ticket labels Nov 28, 2016

PVince81 assigned DeepDiver1975 Dec 12, 2016

michaelstingl added green-ticket and removed blue-ticket labels Dec 20, 2016

PVince81 mentioned this issue Jan 27, 2017

Mysql unit tests fail locally on two openSUSE machines #21161

Closed

cdamken self-assigned this Mar 7, 2017

DeepDiver1975 closed this as completed Mar 8, 2017

SamuAlfageme mentioned this issue Mar 28, 2017

[OC 10.0][OS X] libsync is able to download files with 4 byte-unicode names but upload etc owncloud/client#5661

Closed

3 tasks

SamuAlfageme mentioned this issue Jul 11, 2017

Add the extra configuration to mysql to support 4-byte unicode owncloud-docker/base#4

Closed

lock bot locked as resolved and limited conversation to collaborators Aug 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems with multibyte unicode characters in filenames #4513

Problems with multibyte unicode characters in filenames #4513

georgehrke commented Aug 19, 2013 •

edited by DeepDiver1975

Loading

georgehrke commented Aug 19, 2013

DeepDiver1975 commented Aug 21, 2013

georgehrke commented Aug 22, 2013

DeepDiver1975 commented Aug 22, 2013

georgehrke commented Aug 22, 2013

DeepDiver1975 commented Aug 22, 2013

natschil commented Aug 22, 2013

karlitschek commented Aug 26, 2013

whitehairtong commented Mar 25, 2014

LukasReschke commented Mar 25, 2014

MorrisJobke commented Jul 30, 2015

jaller94 commented Sep 26, 2016 •

edited by peterprochaska

Loading

jaller94 commented Sep 26, 2016 •

edited

Loading

jaller94 commented Sep 26, 2016

PVince81 commented Oct 17, 2016

PVince81 commented Oct 17, 2016 •

edited

Loading

PVince81 commented Dec 12, 2016

michaelstingl commented Dec 20, 2016

cdamken commented Mar 7, 2017

DeepDiver1975 commented Mar 8, 2017

PVince81 commented Mar 8, 2017

lock bot commented Aug 2, 2019

Problems with multibyte unicode characters in filenames #4513

Problems with multibyte unicode characters in filenames #4513

Comments

georgehrke commented Aug 19, 2013 • edited by DeepDiver1975 Loading

Expected behaviour

Actual behaviour

Steps to reproduce

Server configuration

Client configuration

georgehrke commented Aug 19, 2013

DeepDiver1975 commented Aug 21, 2013

georgehrke commented Aug 22, 2013

DeepDiver1975 commented Aug 22, 2013

georgehrke commented Aug 22, 2013

DeepDiver1975 commented Aug 22, 2013

natschil commented Aug 22, 2013

karlitschek commented Aug 26, 2013

whitehairtong commented Mar 25, 2014

LukasReschke commented Mar 25, 2014

MorrisJobke commented Jul 30, 2015

jaller94 commented Sep 26, 2016 • edited by peterprochaska Loading

Steps to reproduce

Preparation

"Share" a White Screen of Death

Share your own root folder

My System

Server

Client

jaller94 commented Sep 26, 2016 • edited Loading

Uploading "..🐦.txt"

Sharing it with another user

jaller94 commented Sep 26, 2016

PVince81 commented Oct 17, 2016

PVince81 commented Oct 17, 2016 • edited Loading

PVince81 commented Dec 12, 2016

michaelstingl commented Dec 20, 2016

cdamken commented Mar 7, 2017

DeepDiver1975 commented Mar 8, 2017

PVince81 commented Mar 8, 2017

lock bot commented Aug 2, 2019

georgehrke commented Aug 19, 2013 •

edited by DeepDiver1975

Loading

jaller94 commented Sep 26, 2016 •

edited by peterprochaska

Loading

jaller94 commented Sep 26, 2016 •

edited

Loading

PVince81 commented Oct 17, 2016 •

edited

Loading