[BUG] Strange behaviour with non breaking spaces

[Beurt]

Bug description

In French, we use a lot of non breaking spaces. And QON has a strange behaviour with them. I attached a test file that contains the two most used non breaking spaces.

test.md

it just contains the following markdown text:

---
title: non breaking space test
headingDivider: 1
--- 


# Test 1 classic NBSP


`+00A0   NO-BREAK SPACE`

+00A0   NO-BREAK SPACE


# Test 2 Narrow


`U+202F   NARROW NO-BREAK SPACE`

U+202F   NARROW NO-BREAK SPACE

When you open the file with a text editor such as kwrite you can see (the first type of) non breaking spaces. You can also use https://apps.timwhitlock.info/unicode/inspect?s= to identify them.

In kwrite (screenshot):

If I put this file in my note folder, and modify it with QON (e.g. adding a word somewhere), the non breaking spaces vanish from the QON saved file (we can check it with kwrite or the aforementioned website).

I am using QON 23.9.2 on Linux KDE Neon (Ubuntu Jammy) and all my scripts are deactivated.

Steps to reproduce

1. copy the example file to the note folder
2. edit it in QON
3. wait for auto-save or save it in QON
4. check for non breaking spaces in the file with another tool (I.e. kwrite)

Can anyone reproduce this?

[Beurt]

More informations:

I made a script that highlight specific characters (such as non breaking spaces) in QON editor panel: https://github.com/qownnotes/scripts/tree/master/characters-highlight

And it is working fine: when I add a non breaking space such as Unicode U+00A0, the character is properly highlighted in the editor panel.

Highlights of the non breaking spaces in the editor panel (Screenshot):

But, not saved in the file (if I check the file with kwrite or other tools the non breaking space is just a basic space).

File saved, reopen with kwrite, where the non breaking spaces are gone (Screenshot):

So, in my understanding, the non breaking spaces are indeed present in the editor panel of QON (hence correctly highlighted), but are not saved in the file on the disk. They are lost during the save mechanism.

[pbek]

The note files are stored as text file if the setting useUNIXNewline is not used:

QOwnNotes/src/entities/note.cpp

Lines 1369 to 1371 in 9d8989e

	if (!useUNIXNewline) {
	flags |= QIODevice::Text;
	}

UTF-8 is used:

QOwnNotes/src/entities/note.cpp

Line 1423 in 9d8989e

out.setCodec("UTF-8");

[pbek]

If you want to try to debug when those characters are lost (will be a bit hard because you can't output them directly) those areas are a good start. Or maybe just removing some code and how the note is stored... Or maybe another character set?

[Beurt]

It seems too tricky for me (I cannot handle C++). But, I found this on the web: https://forum.qt.io/topic/92254/qtextedit-and-unicode-char-no-break-space/4 is it relevant to the current bug?

[pbek]

Thank you, I'll look into it.

[pbek]

noteTextEdit->document()->toRawText() seems to break a lot, starting with the filename... 🙈
No regular expression seems to work on those strings...

[pbek]

I'm giving up on this, I already spend too much time on getting around this. https://doc.qt.io/qt-6/qtextdocument.html#toPlainText says the non-breaking spaces are filtered. But if you use toRawText() all QRegularExpressions are broken.

[Beurt]

I am very sorry for the time spent on that issue. And thank you!

I may be have a last idea. Since toRawText() cannot be used, is it possible to transform U+00A0 into   before toPlainText (or any other operation that transform non breaking spaces into basic spaces) and afterward transform it back to U+00A0? Maybe it is too much?

[pbek]

The problem is that there is no "before toPlainText", toPlainText fetches the text.

[Beurt]

That is unfortunate! I am wondering how the other QT editors (such as kwrite) are managing this.

Unfortunately, I cannot figure out how I can compensate this problem with a script since every modifications (for instance adding automagicaly the missing nbsp) will be wiped when the text is fetched for any further treatment.

😿

[Beurt]

🤔 Reading the toPlainText() doc I am wondering: Is it possible to do a toRawText() then managing/escaping the special characters (U+00A0, U+2029, U+2028) and then doing a toPlainText()? Again, it is maybe impossible or too much for a single missing char.

[pbek]

Only it someone finds out why all QRegularExpressions are not matching any more.

[pbek]

@Waqar144, have you any experience what Kate did?

[Waqar144]

Kate doesn't use QPlainTextEdit, and it stores the text separately and renders it hence it doesn't suffer from this issue.

It would be weird if all regular expressions are not matching anymore, there is nothing special about this text, it's a normal QString after all. So is there any particular regex that doesn't match? Also, it may be enough to just use toRawText when storing the note text on disk.