Incorrect formatting of "right to left" language strings.

[ntoll]

This was originally reported by @Farshidmi on our Gitter chat channel.

The problem is that when creating strings that contain languages that read from right-to-left (like Persian) then Mu (or more likely, QScintilla) doesn't handle things correctly. The picture below shows how the Persian script bleeds out of the quotation marks which delineate the string.

(It's important to note, that the program output is correct though).

What should the editor really look like? Well, @Farshidmi has taken a screen shot of a side-by-side comparison with Notepad++ (which handles things correctly).

[carlosperate]

Just as a quick note, Notepad++ is also based on Scintilla, so perhaps the bug is in the QScintilla or PyQt wrappers.

[ntoll]

Scintilla does not properly support right-to-left languages like Arabic and Hebrew. While text in these languages may appear correct, it is not possible to interact with this text as is normal with other editing components.

See: https://scintilla.sourceforge.io/ScintillaDoc.html

[carlosperate]

Interesting, I wonder how Notepad++ has dealt with this then 🤔

[ntoll]

@carlosperate that's interesting... so why does Notepad++ work?

[ntoll]

@carlosperate hahaha.. OK... we're on the same page. Perhaps the outcome for us is to investigate Notepad++'s work-around.

[carlosperate]

It sounds like Scintilla should perhaps still display the text correctly, I think the main issue is interacting with it (text selection and things like that).

I would search online for the simplest possible scintilla and qscintilla examples possible and see if the text is displayed correctly in one of those.

[ntoll]

OK... I see what's going on. There are different versions of the Scintilla website and they're not in sync. The same document on scintilla.org (updated in 2018) says this:

Scintilla provides only limited experimental support on Windows for right-to-left languages like Arabic and Hebrew. While text in these languages may appear correct, interaction with this text may not work correctly as occurs with other editors.

So it looks like it's definitely a problem they (upstream) are aware of.

[tjguk]

I've just tried it in SciTE which is effectively the reference implementation of Scintilla (and, incidentally, my default editor). On Windows, obviously,

I don't have a particular problem: I used Google translate as the OP suggested and copied/pasted the resulting Persian text into SciTE as print("سلام نړۍ")

I could select and paste into, eg, a Word document apparently correctly. When selecting the selected text reversed, although the resulting paste was correct. Whether that was an error or merely conventional for selecting rtl text, I'm not sure.

[tjguk]

FWIW it's clear that the implementation is not complete; in Word (which I'll take to be correct), the text cursor switches direction in the rtl portion of the text; in SciTE it reverses the underlying string. But I don't know how significant this all is.

[ntoll]

@tjguk aha... thank you. So, if QScintilla is up to date with Scintilla, and we use the latest QScintilla in Mu, then perhaps it should "just work" (tm) on Windows. ;-)

[tjguk]

NB https://www.scintilla.org/ScintillaHistory.html v4.1.0 (June 2018) and a couple of later refs to "bidirectional"

[carlosperate]

Hopefully this might be resolved in the next version of Qscintilla, so we can check the next time the dependencies are updated for the next alpha/beta release.

[dybber]

I just tried with the most recent Qscintilla (2.11.6, November 2020), which also requires us to update to PyQt5.15.2. The issue is still present, so it doesn't seem like an issue we can solve now.

I wanted to try the SciTE editor, to check whether the issue is present in their own editor (https://www.scintilla.org/SciTE.html) However, on Mac OS X it's apparently $42, so I think someone on Win/Linux should test that.

[dybber]

Oh, I see @tjguk already did check that it works on SciTE, then I don't have any idea about how to proceed here.

[ntoll]

This certainly feels like an upstream problem. I'm also unsure what to suggest as a next step. As far as I can tell this has a limited blast radius of Windows QScintilla..??