Segfault on entry of dead_macron

[apoleon]

Hi,

I am hereby forwarding a Debian bug report.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825932

What I’m doing: Try to rename a fleet after the star system ‘Fehū’.

What happens: As soon as I hit the macron deadkey, attempting to type
the ‘ū’, FreeOrion segfaults. The following appears on the console:

terminate called after throwing an instance of 'utf8::invalid_utf8'
what(): Invalid UTF-8
Aborted

The bug submitter also mentioned that:

– acute, grave, circumflex, cedilla, diaresis, tilde, and abovering work
(e.g., é, è, ê, ç, ö, ñ, å)
– micron, ogonek, caron, abovedot, and belowdot don’t cause a crash but do
nothing (e.g., ğ, ę, č, ė, ẹ just print g, e, c, e, e)
– doubleacute causes yet another behaviour: when I type ˝ + o, instead of
ő I get nothing at all

Copy and pasting from a character map doesn't work but this might be another bug.
freeorion.log and freeoriond.log are attached to the Debian bug report. The version is a Git snapshot from 1 May 2016.

[geoffthemedio]

Can you copy-paste into the chat text entry box? Pasting whatever in works for me...

[apoleon]

On 07.06.2016 20:17, Geoff wrote:

Can you copy-paste into the chat text entry box?

Yes, that works for me.

[geoffthemedio]

What keyboard layout has these keys?

Pasting into modal dialogs, such as the rename (whatever) text entry box, is known not to work, due to all hotkeys being disabled whenever a modal window is open.

[apoleon]

On 07.06.2016 20:22, Geoff wrote:

What keyboard layout has these keys?

The original bug submitter wrote that he uses a custom xmodmap because
he does a lot of multilingual work.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825932#15

So it seems he simply added this character to an arbitrary key..

[geoffthemedio]

I don't know how to reproduce this input sequence on Windows, so will have difficulty investigating the issue.

[tkinias]

Hi there – I’m the submitter of the Debian bug.

I wouldn’t call using AltGr + - to get a deadkey macron “arbitrary,” really... it was the most logical thing I could come up with 😄

A little googling seems to reveal that there’s a “English (New Zealand) - Maori” keyboard layout in Windows that maps the macron deadkey to the backtick <`>. See Keyboard setup for macrons. That seems like it might be the easiest way to test on Windows. Unfortunately, I don’t have admin access to a Windows machine to test that myself.

I wasn’t aware that copy/paste worked in chat; I assumed it was universally broken because it doesn’t work in the modal dialog. Testing it now... on my machine, I can paste arbitrary text (including Greek characters and Latin letters with the macron) into chat, but typing the macron deadkey in chat crashes Freeorion just like typing it in a modal dialog.

The irony of this, of course, is that the only reason I ever tried to type a macron in the game at all is that characters with macrons appear in star names in the game, and I was just trying to name one of my ships after one of my star systems!

[tkinias]

@geoffthemedio, is there a bug open for the copy/paste thing? I don’t see one...

[geoffthemedio]

I don't keep careful track of such things. If you can't find it, it's probably not there, or was unobviously named.

I set up a Maori keyboard layout. If I use it now, I get ā but in FreeOrion I just get `a but no crash.

[dbenage-cx]

frame stack
Done in chat window, while holding AltR, crash is immediate on hitting '-', before the desired letter.

[tkinias]

@dbenage-cx, yes, that’s exactly what I see – either in chat window or in a modal dialog (like ship or planet rename). Looks like you’re also on a Linux of some kind.

[dbenage-cx]

using

$ setxkbmap -variant altgr-intl -option -option lv3:ralt_switch

altR+shift+3 results in the same framestack/locals (entry.text.text is different, but 0 is the same)
key combo is for the dead key dead_macron. e.g. altR+shift+3 e = ē

Possible to silently ignore dead keys?

@tkinias I am, though I'm just providing feedback in the hopes it is useful. Using the above, AltR+p displays ö. I did not check them all, but noticed many of the glyphs you noted displayed (when not needing dead keys).

[geoffthemedio]

Does it help if you change the start (wrap the first line) of this function to:

void HumanClientApp::HandleSystemEvents() {
    try {
        SDLGUI::HandleSystemEvents();
    } catch (const std::exception& e) {
        ErrorLogger() << "Error handling system event: " << e.what();
    }

[dbenage-cx]

It prevents the client from crashing, and continues to run

[error] Client : HumanClientApp.cpp:801 : Error handling system event: Invalid UTF-8

However, you can no longer exit the client (through Exit on menu or X on window)

[error] Client : HumanClientApp.cpp:801 : Error handling system event: Quitting SDLGUI normally.

[geoffthemedio]

Try

catch (const utf8::invalid_utf8& e)

(or minor adjustments if that produces a compile error...)

[dbenage-cx]

Allows exit, logs error on dead_macron.
altR+shift+3 a = a (not ā)

Sample of other typed chars:

First two lines are direct key combos: altR+char altR+shift+char
Next 3 are from dead keys: altR+shift+' altR+shift+2 altR+shift+5

If Maori does not have dead keys, Greek Polytonic looks like it has a macron key '-'

[geoffthemedio]

Greek Polytonic isn't in my available layouts, alas.

So does everything exept macron deadkey work, or are those results not as expected?

[dbenage-cx]

macron deadkey is the only one I've found to produce an exception.
I have not found a concise list of dead keys to test against, and the difference between some of the glyphs can be hard to discern.

Mixed results with other dead keys, they either work or are silently ignored without an error.
e.g. altR+shift+9 o = o (not ŏ), no error.

[tkinias]

My experience is that many of the glyphs composed with deadkeys don’t work; they silently get replaced with the base character (e.g., ğ -> g, ő -> o). However, it’s only the macron that crashes the client.

What it looks like to me is that any diacriticals which aren’t present in the old Latin-1 encoding get stripped out – so <á>, <ç>, etc. work, while diacriticals only present in the old Latin-2/3/4 encodings, for example, don’t.

Curiously, I note that some of the characters that don’t work when typed are actually present in stringtables/en.txt and render correctly in the client: e.g., star name <Tàuṛa̋ṛȍ>. This renders correctly both when the star name is generated and when I paste it into chat, but if I try to type <ṛ> in the client I just get .

[geoffthemedio]

Once text is in memory as valid UTF-8, pasted or from a stringtable lookup, it should be rendered fine (assuming fonts support it). Problems with text input / dead keys and such happen in mostly unrelated code, possibly within SDL input handler code, or the code that wraps it.

[geoffthemedio]

Non-functional deadkeys may be an SDL problem:
https://bugzilla.libsdl.org/show_bug.cgi?id=2834
I'm building against 2.0.3 and 2.0.4 is only ~5 months old. If nowhere else, you can check the version you're using in freeorion.log

[tkinias]

Hmm. That could well be. I’ve not encountered problems in any other
applications on Linux, but I can’t off the top of my head think of anything
else I regularly use that uses SDL.

We’re libsdl2.0-2 2.0.4+dfsg2-1 in Debian/Testing, and my freeorion.log
reports the SDL version as 2.0.4.

On Wed, Jun 8, 2016 at 3:09 PM, Geoff notifications@github.com wrote:

Non-functional deadkeys may be an SDL problem:
https://bugzilla.libsdl.org/show_bug.cgi?id=2834
I'm building against 2.0.3 and 2.0.4 is only ~5 months old. If nowhere
else, you can check the version you're using in freeorion.log

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#701 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AQphOVP9Sf1kvcC3RkYjDsaysGWne2Cnks5qJxNfgaJpZM4IwHEr
.

Thanasis Kinias
Doctoral Student, Department of History
Northeastern University
kinias.t@husky.neu.edu

[geoffthemedio]

Should be fixed in that version of SDL, so it's probably something about how FreeOrion is interpreting the textinput messages then.

[tkinias]

FWIW, it appears that the “Invalid UTF8” message that’s dumped to STDERR
when the crash occurs is coming from GG – the utf8::invalid_utf8 exception
is defined in GG/GG/utf8/checked.h. Is it being thrown by something in
GG/src/TextControl.cpp when the deadkey macron is incorrectly identified as
invalid utf8?

Unfortunately I’m not very good at all with C++, so I’m having trouble
following what’s going on there, but based on where that exception comes
from I’m inclined to suspect that this has something to do with GG rather
than SDL...

On Wed, Jun 8, 2016 at 4:39 PM, Geoff notifications@github.com wrote:

Should be fixed in that version of SDL, so it's probably something about
how FreeOrion is interpreting the textinput messages then.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#701 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AQphOX0fvDj024YaFXDzXR_y7a3lD6Zbks5qJyiXgaJpZM4IwHEr
.

Thanasis Kinias
Doctoral Student, Department of History
Northeastern University
kinias.t@husky.neu.edu

[geoffthemedio]

I can't test due to lack of a way to reproduce the exception, but I assume that when a problematic deadkey combination is entered, SDL is generating some events which GG handles here:

https://github.com/freeorion/freeorion/blob/master/GG/src/SDL/SDLGUI.cpp#L692

(either a keyup, keydown, or textinput message). If it's a textinput event, I'm guessing that the crash within utf8::next happens on this line

https://github.com/freeorion/freeorion/blob/master/GG/src/SDL/SDLGUI.cpp#L952

because the text being iterated over isn't valid UTF-8 for some reason. I can't really investigate myself, though.

[dbenage-cx]

For the following tested dead keys, freeorion does not receive a textinput event from SDL:

caron, dot above, dot below, grave, tilde, diaeresis, acute, double acute, cedilla, breve, ring above

An event is received only after a valid corresponding keypress, e.g. acute e = one event for é
(checked by debug breaking at SDLGUI.cpp:949)

It does receive one after a macron dead key (see previous log).
Changing freeorion/GG/GG/utf8/core.h:96

        if (lead < 0x80 || lead == 0xaf)

Results in a ¯ displayed on press of macron dead key, with no exception/error log. macron a = ¯a

Edit: The tested dead keys were with the a character (unless invalid, then e or u). Some either do not to send an event: U+016F (ring above u), or sends one for just the char: U+01CE caron a = a. Not fully tested, looks confined to ranges exceeding Latin-1.

[geoffthemedio]

Try putting breakpoints at line 694 and 702:
https://github.com/freeorion/freeorion/blob/master/GG/src/SDL/SDLGUI.cpp#L694
Or even 689, and then seeing what events are produced by using various deadkey + letter combinations. Or perhaps put something like this right before the switch:

        if (event.key.keysym.scancode != 0) {
            std::cout << "event type: " << event.type << "  key.keysym.scancode:" << event.key.keysym.scancode
                      << "  text: " << event.text.text << std::endl;
        }

[dbenage-cx]

altR+shift+3 e (macron e)

event type: 768  key.keysym.scancode:230  text: 
event type: 768  key.keysym.scancode:225  text: 
event type: 768  key.keysym.scancode:32  text: 
event type: 771  key.keysym.scancode:65108  text: �
event type: 769  key.keysym.scancode:32  text: 
event type: 769  key.keysym.scancode:230  text: 
event type: 769  key.keysym.scancode:225  text: 
event type: 768  key.keysym.scancode:8  text: 
event type: 771  key.keysym.scancode:101  text: e
event type: 769  key.keysym.scancode:8  text: 

altR+shift+2 u (double acute u)

event type: 768  key.keysym.scancode:230  text: 
event type: 768  key.keysym.scancode:225  text: 
event type: 769  key.keysym.scancode:230  text: 
event type: 769  key.keysym.scancode:225  text: 
event type: 771  key.keysym.scancode:117  text: ü
event type: 512  key.keysym.scancode:192  text:

Need a full listing?

[geoffthemedio]

Looks to me like SDL doesn't know what to do with your macron deadkey event... It appears to be sending just 'e' and not a modified character as occurs with the composed 'ü'.

[geoffthemedio]

copy/cut/paste hotkeys should now work in modal dialogs.

[geoffthemedio]

As the segfault is fixed, and I don't know how to do anything about the other nonfunctional deadkeys, I will close this.