Found some Notes on the Server with "ZERO WIDTH NO-BREAK SPACE" character encoding, its encoding is FEFF

[Githopp192]

Steps to reproduce

Cannot reproduce

Expected behaviour

no hidden characters on the Server

Have got hidden Files on a Nextcloud Server can break it !!!

Actual behaviour

some Notes have this hidden character -->

drwxr-xr-x 2 apache apache 4 Oct 23 19:28 .
-rw-r--r-- 1 apache apache 0 Oct 23 19:28 file.txt
drwxr-xr-x 11 apache apache 11 Sep 16 17:23 ..
-rw-r--r-- 1 apache apache 1681 Dec 18 2019 file.txt

Windows Desktop sync will not work when resyncing such files on the server (it's some kind of server name corruption !)

Server configuration detail

Operating system: Linux 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64

Webserver: Apache (fpm-fcgi)

Database: mysql 10.3.17

PHP version:

7.4.11
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, intl, json, ldap, exif, mysqlnd, PDO, Phar, posix, shmop, SimpleXML, sockets, sodium, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlwriter, xsl, mysqli, pdo_mysql, pdo_sqlite, xmlreader, apcu, igbinary, imagick, msgpack, smbclient, zip, memcached, redis, libsmbclient, Zend OPcache

Nextcloud version: 19.0.4 - 19.0.4.2

Updated from an older Nextcloud/ownCloud or fresh install: 19.0.3

Where did you install Nextcloud from: Nextcoud source

Signing status

Array
(
)

List of activated apps

Enabled:
 - accessibility: 1.5.0
 - activity: 2.12.1
 - analytics: 2.5.0
 - announcementcenter: 3.8.1
 - apporder: 0.11.0
 - audioplayer: 2.12.0
 - audioplayer_editor: 0.3.0
 - audioplayer_sonos: 1.2.0
 - bookmarks: 3.4.4
 - bruteforcesettings: 2.0.1
 - calendar: 2.0.4
 - circles: 0.19.7
 - cloud_federation_api: 1.2.0
 - comments: 1.9.0
 - contacts: 3.4.1
 - contactsinteraction: 1.0.0
 - cookbook: 0.7.6
 - cospend: 1.0.5
 - data_request: 1.6.0
 - dav: 1.15.0
 - deck: 1.1.2
 - dicomviewer: 1.2.2
 - documentserver_community: 0.1.8
 - drawio: 0.9.7
 - end_to_end_encryption: 1.5.2
 - event_update_notification: 1.0.2
 - extract: 1.2.4
 - federatedfilesharing: 1.9.0
 - federation: 1.9.0
 - files: 1.14.0
 - files_3d: 0.3.2
 - files_antivirus: 3.0.0
 - files_automatedtagging: 1.9.0
 - files_downloadactivity: 1.8.0
 - files_markdown: 2.3.1
 - files_mindmap: 0.0.23
 - files_pdfviewer: 1.8.0
 - files_photospheres: 1.19.1
 - files_rightclick: 0.16.0
 - files_sharing: 1.11.0
 - files_trashbin: 1.9.0
 - files_versions: 1.12.0
 - files_videoplayer: 1.8.0
 - firstrunwizard: 2.8.0
 - flowupload: 1.0.0
 - forms: 2.0.4
 - gpxedit: 0.0.13
 - gpxmotion: 0.0.11
 - gpxpod: 4.2.2
 - groupfolders: 7.1.0
 - impersonate: 1.6.1
 - issuetemplate: 0.7.0
 - logreader: 2.4.0
 - lookup_server_connector: 1.7.0
 - maps: 0.1.6
 - nextcloud_announcements: 1.8.0
 - notes: 3.6.4
 - notifications: 2.7.0
 - oauth2: 1.7.0
 - onlyoffice: 6.1.0
 - passman: 2.3.6
 - password_policy: 1.9.1
 - photos: 1.1.0
 - polls: 1.5.4
 - privacy: 1.3.0
 - provisioning_api: 1.9.0
 - quicknotes: 0.6.1
 - quota_warning: 1.8.0
 - rainloop: 7.0.3
 - ransomware_protection: 1.7.0
 - recommendations: 0.7.0
 - serverinfo: 1.9.0
 - settings: 1.1.0
 - sharebymail: 1.9.0
 - socialsharing_diaspora: 2.1.0
 - socialsharing_email: 2.1.0
 - socialsharing_facebook: 2.1.0
 - socialsharing_twitter: 2.1.0
 - spreed: 9.0.5
 - support: 1.2.1
 - suspicious_login: 3.2.1
 - systemtags: 1.9.0
 - tasks: 0.13.5
 - terms_of_service: 1.5.2
 - text: 3.0.1
 - theming: 1.10.0
 - timemanager: 0.1.4
 - twofactor_backupcodes: 1.8.0
 - twofactor_totp: 5.0.0
 - updatenotification: 1.9.0
 - video_converter: 0.1.5
 - viewer: 1.3.0
 - workflow_ocr: 1.19.1
 - workflow_pdf_converter: 1.4.0
 - workflow_script: 1.4.0
 - workflowengine: 2.1.0
Disabled:
 - admin_audit
 - breezedark
 - dashboard
 - drop_account
 - encryption
 - external
 - files_accesscontrol
 - files_external
 - files_fulltextsearch
 - fulltextsearch
 - fulltextsearch_elasticsearch
 - joplin
 - jsloader
 - passwords
 - registration
 - richdocuments
 - socialsharing_googleplus
 - survey_client
 - user_ldap
 - weather

Configuration (config/config.php)

{
    "memcache.local": "\\OC\\Memcache\\APCu",
    "filelocking.enabled": true,
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "dbindex": "***REMOVED SENSITIVE VALUE***",,
        "timeout": "***REMOVED SENSITIVE VALUE***",,
        "password": "***REMOVED SENSITIVE VALUE***"
    },
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
       "***REMOVED SENSITIVE VALUE***",
        "***REMOVED SENSITIVE VALUE***",
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/www.swiss7cloud.ch",
    "htaccess.RewriteBase": "\/",
    "overwriteprotocol": ""***REMOVED SENSITIVE VALUE***",",
    "dbtype": "mysql",
    "version": "19.0.4.2",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "maintenance": false,
    "theme": "",
    "loglevel": 0,
    "updater.release.channel": "stable",
    "auth.bruteforce.protection.enabled": true,
    "check_for_working_htaccess": true,
    "data-fingerprint": ""***REMOVED SENSITIVE VALUE***",",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "tls",
    "mail_smtpauth": 1,
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpport": ""***REMOVED SENSITIVE VALUE***",",
    "session_lifetime": 1200,
    "session_keepalive": false,
    "logtimezone": ""***REMOVED SENSITIVE VALUE***",,
    "logfile": "\/media\/log\/nextcloud.log",
    "knowledgebaseenabled": false,
    "log_rotate_size": 3145728,
    "mail_sendmailmode": "smtp",
    "app_install_overwrite": [
        "passman",
        "dicomviewer",
        "radio"
    ]
}

Are you using external storage, if yes which one: no

Are you using encryption:

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36 Edg/86.0.622.48

Operating system: Windows 10

Logs

Web server error log

Insert your web server log here 

[korelstar]

The disruptive character (0xFEFF) is Byte Order Mark (BOM) for UTF-16. Therefore, I think the source for the problem could be an character set encoding problem on the server side (operating system). Since you are not able to reproduce the issue, I'm not sure what we can do, here.

Please try to fix the issue by renaming the note (click on three-dots button for that note in the app-navigation an then "Rename").

[Githopp192]

hm .. i don't think there is an issue on the server or Operating system:

Server Settings -->

php -i | grep -i utf -->

default_charset => UTF-8 => UTF-8
LANG => en_US.UTF-8
$_SERVER['LANG'] => en_US.UTF-8

echo $LANG

en_US.UTF-8

I did make a find/grep over the whole Cloud Files (there a more than 1 Millions of Files !).

i found only BOM Files into the Notes-Application; this could be an indication, that there is no Server- or OS Issue, otherwise other files would also be affected, but this is not the fact -->

-rw-r--r-- 1 apache apache 1681 Dec 18 2019 File1.txt
-rw-r--r-- 1 apache apache 1679 Dec 18 2019 File1.txt.v1576602674
-rw-r--r-- 1 apache apache 1460 Dec 17 2019 File1.txt.v1574169213
-rw-r--r-- 1 apache apache 662 Nov 8 2019 File1.txt.v1571761220
-rw-r--r-- 1 apache apache 105 Oct 22 2019 File1.txt.v1555106119
-rw-r--r-- 1 apache apache 54291 Feb 14 2020 File2.txt.v1581690331
-rw-r--r-- 1 apache apache 53643 Feb 14 2020 File2.txt.v1557593983
-rw-r--r-- 1 apache apache 54291 May 11 2019 File2.txt.v1518827783
-rw-r--r-- 1 apache apache 54291 Feb 14 2020 File2.txt

it concerns both the year 2019 and the year 2020.

The total summary of the Notes Files are about 100; so this issue affect about 10% of the files (including the file versions).

(of course .. the renaming is working from the GUI or on the operations system)

[korelstar]

But without file versions, there are only two files affected?

The important question is: what's the difference between these two files and the other ones which are working? How did you created them (e.g. Notes app, Files app, Android app, on desktop (which operating system?) using file synchronisation, ...)? How was the title of these note set (e.g. automatically by the notes app or manually)? If we want to fix this, we need to find out which component you've used creates UTF-16 strings.

[Githopp192]

made some tests - i.e. create a note with windows 10 notepad.exe .. then transferred this note to nc-explorer .. file was sychronized .. and the file has got the correct encoding - so no problem from there i think

The Difference is the BOM, see here -->

https://stackoverflow.com/questions/2223882/whats-the-difference-between-utf-8-and-utf-8-without-bom#:~:text=The%20UTF%2D8%20BOM%20is,8%2C%20the%20BOM%20is%20unnecessary.

Normal Files on the Operating system level looks like this:

file1.txt

Bom Files looks like this (you only see the encoding, when you do a "ls -alt > testfile".
When doing a "normal" "ls -alt" you do not see the encoding.

Files with encoding looks like that -->

<feff>file1.txt

I'm quite sure, that i did create those files with issues on the NC Web GUI .. i did copy/paste the content from another web page.

Just now i did the same (copy/paste a IE Page and paste that to "notes" .. but encoding is correct (no at the beginning).

About your question about the "title" for all these notes, which have got encoding issues, the title was set automatically.

Of course you're right .. we need to find the culprit :-)

[Githopp192]

i think we come closer to the reason for ..
i just wanted to remame the corrupt files via Notes and saw the guilty char

have a look here ... this is a pic from it -->

you see the little orange circle .. this is the hidden char .. and the NC Notes Apps does take the first line as a Title, when you do not choose it manually.

Now i did copy this guiltiy char and paste it here -->

https://www.soscisurvey.de/tools/view-chars.php

After deleting this char, the hidden UTF-8/BOM encoding letter on the fils is gone - gone on NC-GUI and gone on the OS-Level.

I think solution is .. remove the hidden BOM chars from the Note, when saving it from the GUI to the OS Level