Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Windows] test_winconsoleio failures #82506

vstinner opened this issue Sep 30, 2019 · 7 comments

[Windows] test_winconsoleio failures #82506

vstinner opened this issue Sep 30, 2019 · 7 comments
3.8 only security fixes 3.9 only security fixes OS-windows


Copy link

BPO 38325
Nosy @terryjreedy, @pfmoore, @tjguk, @zware, @eryksun, @zooba
  • bpo-38325: Fix test_winconsoleio #18448
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-02-10.23:59:42.594>
    created_at = <Date 2019-09-30.13:20:15.398>
    labels = ['3.8', '3.9', 'OS-windows']
    title = '[Windows] test_winconsoleio failures'
    updated_at = <Date 2020-09-06.10:10:37.114>
    user = '' fields:

    activity = <Date 2020-09-06.10:10:37.114>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-02-10.23:59:42.594>
    closer = 'vstinner'
    components = ['Windows']
    creation = <Date 2019-09-30.13:20:15.398>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 38325
    keywords = ['patch']
    message_count = 7.0
    messages = ['353571', '353613', '353616', '353627', '361758', '361759', '376450']
    nosy_count = 6.0
    nosy_names = ['terry.reedy', 'paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower']
    pr_nums = ['18448']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = ''
    versions = ['Python 3.8', 'Python 3.9']

    Copy link
    Member Author

    On Windows 10 version 1903, test_winconsoleio even when run from cmd.exe console.

    C:\vstinner\python\3.8>python -m test -v test_winconsoleio
    Running Release|x64 interpreter...
    == CPython 3.8.0b4+ (heads/pr/16490:8a204fd07c, Sep 30 2019, 14:29:53) [MSC v.1916 64 bit (AMD64)]
    == Windows-10-10.0.18362-SP0 little-endian
    == cwd: C:\vstinner\python\3.8\build\test_python_1616
    == CPU count: 2
    == encodings: locale=cp1252, FS=utf-8
    Run tests sequentially
    0:00:00 [1/1] test_winconsoleio
    test_abc (test.test_winconsoleio.WindowsConsoleIOTests) ... ok
    test_conin_conout_names (test.test_winconsoleio.WindowsConsoleIOTests) ... ok
    test_conout_path (test.test_winconsoleio.WindowsConsoleIOTests) ... ok
    test_ctrl_z (test.test_winconsoleio.WindowsConsoleIOTests) ... Ä^Z
    test_input (test.test_winconsoleio.WindowsConsoleIOTests) ... abc123
    A͏B ﬖ̳AA̝
    test_open_fd (test.test_winconsoleio.WindowsConsoleIOTests) ... ok
    test_open_name (test.test_winconsoleio.WindowsConsoleIOTests) ... ok
    test_partial_reads (test.test_winconsoleio.WindowsConsoleIOTests) ... �ERROR
    test_partial_surrogate_reads (test.test_winconsoleio.WindowsConsoleIOTests) ... �ERROR
    test_write_empty_data (test.test_winconsoleio.WindowsConsoleIOTests) ... ok

    ERROR: test_input (test.test_winconsoleio.WindowsConsoleIOTests)

    Traceback (most recent call last):
      File "C:\vstinner\python\3.8\lib\test\", line 148, in test_input
      File "C:\vstinner\python\3.8\lib\test\", line 135, in assertStdinRoundTrip
        actual = input()
    OSError: [WinError 87] Paramètre incorrect

    ERROR: test_partial_reads (test.test_winconsoleio.WindowsConsoleIOTests)

    Traceback (most recent call last):
      File "C:\vstinner\python\3.8\lib\test\", line 161, in test_partial_reads
        b =
    OSError: [WinError 87] Paramètre incorrect

    ERROR: test_partial_surrogate_reads (test.test_winconsoleio.WindowsConsoleIOTests)

    Traceback (most recent call last):
      File "C:\vstinner\python\3.8\lib\test\", line 178, in test_partial_surrogate_reads
        b =
    OSError: [WinError 87] Paramètre incorrect

    Ran 10 tests in 0.013s

    FAILED (errors=3)
    test test_winconsoleio failed
    test_winconsoleio failed

    == Tests result: FAILURE ==

    1 test failed:

    Total duration: 62 ms
    Tests result: FAILURE

    @vstinner vstinner added 3.8 only security fixes 3.9 only security fixes OS-windows labels Sep 30, 2019
    Copy link

    eryksun commented Sep 30, 2019

    Apparently handling non-BMP codes is broken in recent builds of the new console in Windows 10. I see this problem in build 18362 as well. It seems there have been updates that have changed the naive way the console used to handle surrogate codes as just regular UCS-2 codes, and this has disrupted the UTF-16 wide-character API in several ways. This is probably related to the new support for virtual-terminal emulation and pseudoconsoles, since supporting a UTF-8 stream interface has required significant redesign of the console backend.

    Low-level ReadConsoleInputW and WriteConsoleInputW still work, but high-level ReadConsoleW now fails if it encounters a non-BMP surrogate pair, i.e. at least two key-event records with the non-BMP character encoded as a UTF-16 surrogate pair. It can be more than two input records depending on the source of input -- WriteConsoleInputW vs pasting from the clipboard -- in terms of KeyDown/KeyUp events or an Alt+Numpad sequence.

    There are issues with reading from screen buffers as well. WriteConsoleW can still successfully write non-BMP characters, and these can be copied from the console fine. But ReadConsoleOutputCharacterW can no longer read them. This used to work, but now it 'succeeds with 0 characters read if the screen-buffer region contains a non-BMP character. I checked the lower-level ReadConsoleOutputW function, and it's behaving differently now. It used to read a non-BMP character as two CHAR_INFO records containing the surrogate pair codes, but now it reads a non-BMP character as a single CHAR_INFO record containing a replacement character U+FFFD.

    I suppose we need to skip testing non-BMP and surrogate codes if the Windows version is (10, 0, 18362) and above.

    Also, _testconsole needs to support FlushConsoleInputBuffer. Every test that calls _testconsole.write_input should be isolated with a try/finally that flushes the input buffer at the end. For example:

        write_input(raw, 'spam')
            actual = input()

    If reading fails, 'spam' will be flushed from the input buffer.

    Copy link
    Member Author

    Until a fix is found, can we mark the tests as "known to fail" on a specific Windows version? (So test_winconsoleio doesn't fail anymore.)

    Copy link

    eryksun commented Oct 1, 2019

    Does test_partial_reads fail for you when run separately? If so, it's for a different reason. Otherwise, there may have been text left in the input buffer from test_input that led to the failure, which is a separate problem that needs to be addressed via FlushConsoleInputBuffer.

    Until a fix is found, can we mark the tests as "known to fail" on a
    specific Windows version? (So test_winconsoleio doesn't fail anymore.)

    Maybe also split out the non-BMP case in test_input to a separate test_non_bmp_input case that's skipped or expected to fail if sys.getwindowsversion() >= (10, 0, 18362).

    As to a fix, there's nothing we can do in Python. An issue can be opened at I wouldn't expect ReadConsoleOutputW to be fixed. But they should be able to fix ReadConsoleW and ReadConsoleOutputCharacterW.

    Copy link
    Member Author

    New changeset 038770e by Victor Stinner in branch 'master':
    bpo-38325: Skip non-BMP tests of test_winconsoleio (GH-18448)

    Copy link
    Member Author

    I "fixed" the issue by skipping broken tests. If anyone wants to fix the underlying issue: go ahead and open an issue, it would be way better with a PR to fix these tests!

    In the meanwhile, skipping broken tests help to detect regressions (new issues).

    Copy link

    test_consoleio now fails and hangs for me. bpo-41729.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    3.8 only security fixes 3.9 only security fixes OS-windows
    None yet

    No branches or pull requests

    3 participants