curses.panel misbehaves when drawing `wchar_t` characters

[blex-max]

Bug report

Bug description:

import curses
import curses.panel as panels

def make_panel(y_mag, x_mag, yO, xO):
 win = curses.newwin(y_mag, x_mag, yO, xO)
 panel = panels.new_panel(win)
 return win, panel


def mainloop(stdscr):
    stdscr.clear()
    stdscr.nodelay(True)
    curses.curs_set(0)
    w1, p1 = make_panel(3, 20, 0, 0)
    w2, p2 = make_panel(3, 20, 0, 0)
    panels.update_panels()
    w1.addstr(0, 0, '-----------')
    # w2.addstr(0, 5, 'd')  # works
    w2.addstr(0, 5, '港d')  # works (kind of)
    # w2.addstr(0, 5, '港')  # fails to draw the contents of w1 to the right of the kanji character
    run = True
    while run:
        w1.refresh()
        w2.refresh()
        panels.update_panels()

        c = stdscr.getch()
        if c == 113:  # q
            run = False


if __name__ == '__main__':
    curses.wrapper(mainloop)

When using kanji charcters in panels, the content to the right of the character in the same row is not displayed. If the kanji is followed by a regular ascii character, everything displays as normal. The doc is a bit limited, so I'm not entirely clear if the panel should be showing anything at all from the panel below, but either way, this doesn't seem to be working as intended. Thanks for any insight :)

CPython versions tested on:

3.13

Operating systems tested on:

macOS

[ronaldoussoren]

I can reproduce this using the python.org installer. One thing I noticed: HAVE_NCURSESW is false for this build.

That said, a local build that did have HAVE_NCURSESW set to true also has this problem.

An ancient 2.7 build that I have lying around does work correctly with some adjustments to the script, but that's also linked to an ancient versions of ncurses.

This is clearly unwanted behaviour, but at this point it is not yet clear if this is a bug in Python or the curses library.

[picnixz]

From the manpage:

If the locale is not initialized, the library assumes that characters are printable as in ISO-8859-1, to work with certain legacy programs. You should initialize the locale and not rely on specific details of the library when the locale has not been setup.

So, without calling setlocale(LC_ALL, ""); before initscr, we're left with some issues with ncurses. Does it work if you call locale.setlocale(LC_ALL, "") before wrapper()?

EDIT: ok it doesn't work for me :D

[picnixz]

A bit more information:

• I changed w1.addstr(0, 0, '0123456789abcdefhijklmnopqrst')
• I added locale.setlocale(LC_ALL, "") before the call to curses.wrapper().
• The output is:

01234港
lmnopqrst

Since the panel is 20 characters wide, we're only losing the 14 characters "6789abcdefhijk" and then the second panel line is because it was wrapped. So everything on the rightn of a wide char is actually removed. I'm wondering if it's because it's null terminated.

[picnixz]

We have a comment that says (emphasis mine):

> Paint the string str at (y, x) with attributes attr, overwriting anything previously on the display.

I'm not sure if this is relevant, but does it mean that we're actually overwriting anything coming after and that's how it's supposed to work? OTOH, in _curses_window_addstr_impl, I've hardcode the wide char to print, and we still have the same "issue", so it's likely not an issue of the conversion to wide chars. NVM, this was not relevant according to the ncurses docs.

In addition, considering we don't do anything else except calling mvwaddwstr, I think this would be an upstream curses issue =/

The fact that w2.addstr(0, 5, 'X') and w2.addstr(0, 5, '港X') work is weird. I've tested w2.addstr(0, 5, 'X港') and it fails. It seems it's really when the string ends with a wide character. So I do think it's an upstream issue.

[ned-deily]

I can reproduce this using the python.org installer. One thing I noticed: HAVE_NCURSESW is false for this build

(FTR, there is an issue with current 3.12 and 3.13 python.org installers for macOS that has caused them to inadvertently link with the system-supplied ncurses libraries. This problem has been corrected in the 3.14.0 alpha 5 preview with the upgrade to ncurses 6.5 which will also appear in the next releases of 3.12 (3.12.10) and 3.13 (3.13.3). But, as noted above, that doesn't seem to be a factor here.)

[ronaldoussoren]

So I do think it's an upstream issue.

That's likely. I haven't tried to reproduce the issue in C yet though.

[ronaldoussoren]

The code below should be a reproducer in C, which for me gives the same output:

-----d-----
-----港d---
-----港

This happens both when linking with the system ncurses (with a minor change to the include lines) and when building and linking with a local build of ncursesw version 6.4.

The code below is quick&dirty, but does show that this is an upstream bug. That said, I'm not a curses expert.

#include <ncursesw/ncurses.h>
#include <panel.h>
#include <locale.h>
#include <unistd.h>

int main(void)
{
    setlocale(LC_ALL, "");

    WINDOW* stdscr = initscr();

    noecho();
    cbreak();

    clear();
    nodelay(stdscr, 1);

    curs_set(0);


    WINDOW* w1 = newwin(3, 20, 0, 0);
    PANEL* p1 = new_panel(w1);

    WINDOW* w2 = newwin(3, 20, 0, 0);
    PANEL* p2 = new_panel(w2);
    update_panels();

    (void)mvwaddstr(w1, 0, 0, "-----------");
    (void)mvwaddstr(w1, 1, 0, "-----------");
    (void)mvwaddstr(w1, 2, 0, "-----------");

    (void)mvwaddstr(w2, 0, 5, "d");  // works
    (void)mvwaddstr(w2, 1, 5, "港d");  // works (kind of)
    (void)mvwaddstr(w2, 2, 5, "港");  // fails to draw the contents of w1 to the right of the kanji character

    (void)wrefresh(w1);
    (void)wrefresh(w2);
    update_panels();

    sleep(5);
    echo();
    nocbreak();
    endwin();
}

[blex-max]

Thanks for all the investigation so far - as the problem is upstream, what's the best way to send this issue in the appropriate direction?

[ned-deily]

@blex-max, according to the project's website, here is their process for reporting bugs. If you want to pursue it, it would probably be good to include a simple reproducer, like @ronaldoussoren's C program (if he doesn't mind). Since this appears to be an upstream issue, let's close this.