Inconsistent handling of Unicode (emoji using variation selectors) between curses.window.addstr and curses.window.insstr

[zahlman]

Bug report

Bug description:

As background: the glyph ❤️ actually consists of two characters: ❤ U+2764 HEAVY BLACK HEART, followed by the unprintable U+FE0F VARIATION SELECTOR-16.

Test script:

import curses


def main(screen):
    screen.clear()
    curses.curs_set(0)
    curses.use_default_colors()
    screen.addstr(0, 0, '\u2764\ufe0ftest')
    screen.addstr('\u2764\ufe0ftest')
    screen.insstr(1, 0, '\u2764\ufe0ftest')
    screen.insstr('\u2764\ufe0ftest')
    try:
        while screen.getch() == -1:
            pass
    except KeyboardInterrupt: # suppress the traceback
        return


if __name__ == '__main__':
    curses.wrapper(main)

The exact appearance of the result is admittedly terminal-dependent, as not all terminals will support rendering the variant-selected red heart. However, the result from .insstr is clearly different in every terminal I could readily test. While .addstr allows the heart emoji to be rendered in terminals that support it (such as gnome-terminal), .insstr only ever displays the ❤ symbol. Further, the spacing is different. It is as though the variation selector had been quietly replaced with a space.

This affects all current versions of Python as far as I can tell, and has probably always affected Python 3 (curses in Python 2 doesn't seem to support Unicode at all).

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[ZeroIntensity]

Without looking too deep, there are two possibilites here:

1. It's a bug with GNU's ncurses, which our module simply wraps. If that's the case, you should report this to the bug-ncurses mailing list. I think this is probably the most likely culprit.
2. curses is running on a system without support for wide-chars in ncurses (see the HAVE_NCURSESW macro). In that case, I doubt there's anything we can do, because the ncurses functions that we call only take ASCII strings.

[zahlman]

With further investigation I'm fairly convinced the issue is indeed in the C library. For example it can be reproduced that way: https://stackoverflow.com/questions/69804246

So I went ahead and reported it on the ncurses mailing list... I'll see what happens from there.

[ZeroIntensity]

I'm going to close this for now, then. Feel free to tag me for reopening if upstream isn't cooperative.