gh-111201: Support pyrepl on Windows #119559

DinoV · 2024-05-25T18:41:49Z

This adds Windows support for the new pyrepl.

Support is a mix of vt100 codes, enabling the vt100 codes on the Windows console, and Windows console APIs.

Minimum version required in Windows 10.

Issue: A new Python REPL #111201

📚 Documentation preview 📚: https://cpython-previews--119559.org.readthedocs.build/

Fix scrolling into input that has scrolled out of the buffer

Lib/test/test_pyrepl/test_windows_console.py

Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com>

zadjii-msft

Hey so sorry for coming out of nowhere to leave notes on this. I'm super excited to see this land! I took a spin through this PR just to see if I could find anything egregious in here from a Windows Console standpoint, and nothing really popped out.

I obviously don't really know the pyrepl codebase closely, so my review probably isn't the most valuable. I left my own notes-to-self with a "📝:", mostly just for things I found confusing. Definitely feel free to ignore those.

overall I'd say: I'm super glad y'all aren't going to be using COOKED_READ in the console anymore 🎉

zadjii-msft · 2024-05-30T20:52:56Z

Lib/_pyrepl/windows_console.py

+        info = CONSOLE_SCREEN_BUFFER_INFO()
+        if not GetConsoleScreenBufferInfo(OutHandle, info):
+            raise WinError(GetLastError())
+        return info.dwCursorPosition.X, info.dwCursorPosition.Y


📝: This is relative to the origin of the buffer, not of the viewport. is this supposed to be viewport relative in the unix version?

Lib/_pyrepl/windows_console.py

zadjii-msft · 2024-05-30T21:09:22Z

Lib/_pyrepl/windows_console.py

+
+            return Event(evt="key", data=key, raw=rec.Event.KeyEvent.uChar.UnicodeChar)
+
+    def push_char(self, char: int | bytes) -> None:


I suppose I don't totally understand how this is supposed to be used, but there is WriteConsoleInput which sounds similar

This is specifically about push_char? I don't think it's actually used but it does seem to technically be public API surface. The unix version calls this on inputted chars.

zadjii-msft · 2024-05-30T21:38:04Z

Lib/_pyrepl/windows_console.py

+
+        self.__posxy = 0, 0
+        self.__gone_tall = 0
+        self.__offset = 0


📝: the unix version also:

enables bracketed paste

does a smkx (\x1b[?1h)

gets the original size (to restore on exit?)

I'm pretty sure the console supported ?1h back to RS1, and bracketed paste as of microsoft/terminal#15155 (not sure exactly where in Windows 11 that gets enabled)

You're probably fine continuing to not do them here, but they could maybe get added later.

fwiw, this REPL is only activated on builds >= 10586
https://github.com/python/cpython/pull/119559/files#diff-45ae56fc192319dbdbe6aca3aa4c7abd31a4a5a38b775805249ba765c9476976R8

pablogsal · 2024-05-30T23:43:30Z

Thanks for your notes @zadjii-msft!

One problem we are still having I think is that resizing the terminal when there are wide Unicode characters such as emojis in many lines doesn't work (the characters don't get resposition after a resize makes the lines warp around and back to normal). This looks like it's a windows only problem because the unix terminal handles that correctly.

I think @ambv or @DinoV may be able to provide more information.

ambv · 2024-05-31T07:48:46Z

Unicode input and emojis work now. Amazing progress! I'm landing this, we can add further improvements like bracketed paste in future updates. Thanks, @DinoV! ✨ 🍰 ✨

bedevere-bot · 2024-05-31T08:08:19Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD14 3.x has failed when building commit 0d07182.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1232/builds/2462) and take a look at the build logs.
Check if the failure is related to this commit (0d07182) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1232/builds/2462

Failed tests:

test_pyrepl

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/libregrtest/single.py", line 181, in _runtest_env_changed_exc
    _load_run_test(result, runtests)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/libregrtest/single.py", line 138, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/libregrtest/single.py", line 91, in regrtest_runner
    test_result = test_func()
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/libregrtest/single.py", line 135, in test_func
    return run_unittest(test_mod)
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/libregrtest/single.py", line 35, in run_unittest
    raise Exception("errors while loading tests")
Exception: errors while loading tests


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/unittest/loader.py", line 396, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/unittest/loader.py", line 339, in _get_module_from_name
    __import__(name)
    ~~~~~~~~~~^^^^^^
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/test/test_pyrepl/test_windows_console.py", line 5, in <module>
    from _pyrepl.windows_console import (
    ...<5 lines>...
    )
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/_pyrepl/windows_console.py", line 30, in <module>
    import ctypes
  File "/home/buildbot/buildarea/3.x.opsec-fbsd14/build/Lib/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

ambv · 2024-05-31T09:28:33Z

Buildbot failure related. Fixing now.

miss-islington-app · 2024-05-31T12:43:17Z

Thanks @DinoV for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2024-05-31T12:43:17Z

Thanks @DinoV for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

(cherry picked from commit 0d07182) Co-authored-by: Dino Viehland <dinoviehland@gmail.com> Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>

bedevere-app · 2024-05-31T12:43:29Z

GH-119850 is a backport of this pull request to the 3.13 branch.

zadjii-msft · 2024-05-31T14:55:16Z

One problem we are still having I think is that resizing the terminal when there are wide Unicode characters such as emojis in many lines doesn't work (the characters don't get resposition after a resize makes the lines warp around and back to normal). This looks like it's a windows only problem because the unix terminal handles that correctly

Oh that for sure sounds like a us bug. We've been making pretty dramatic improvements to the unicode / emoji support over the last couple years, so I'm guessing you're seeing different behavior even depending on what version of Windows & the Terminal you're running.

If you've got specific bugs on your tracker for those, point me at 'em, and I can get the team to double check them ☺️

EDIT: derp I only read one mail, then replied to it before I saw #119559 (comment).

DHowett · 2024-05-31T15:15:59Z

Lib/_pyrepl/__main__.py

+if sys.platform != "win32":
+    CAN_USE_PYREPL = True
+else:
+    CAN_USE_PYREPL = sys.getwindowsversion().build >= 10586  # Windows 10 TH2


This merged as-is which is OK, but I wanted to chip in...

There is a minimal risk of cutting out terminals that would support pyrepl on earlier versions of Windows. The "most correct" (also most soapboxy, sorry!) way to determine support for this is to just try to set ENABLE_VIRTUAL_TERMINAL_PROCESSING and only fall back to basic console APIs if it fails. :)

@DHowett sounds like an improvement we can still make! Care to PR this?

I can try - at the very least, it would be a great opportunity for me to learn how to build and deploy Python 🙂

@DHowett see here and let me know if anything's missing:
https://devguide.python.org/getting-started/setup-building/#windows

(cherry picked from commit 0d07182) Co-authored-by: Dino Viehland <dinoviehland@gmail.com> Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>

Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>

DinoV and others added 30 commits May 16, 2024 18:58

Windows repl support

1061d22

Arrow key support

fa0f538

Make backspace clear char

6f35100

Fix missing newline after input

7772787

Make insert work

aced5ae

Fix delete in middle

fa3815f

Fix crash on invalid command key

b30b105

More fixes

df262c4

Colorize

6e57316

Use constants

18ecc2e

Use UnicodeChar

60525eb

Simplify

af79a17

Fix scrolling on input which is longer than screen/scrollback

5f99256

fix pager typo and refactor some unused branches (python#41)

bdff535

Implement screen clear for Windows (python#42)

88d64f5

Fix word wrap not being enabled in Windows Terminal

8a74306

Fix scrolling into input that has scrolled out of the buffer

Fix culmitive errors in wrapping as lines proceed

38e9c58

Fix issues with inputs longer than a single line

9cddace

Resize WIP

fc4efee

set compat based on Windows build version

564e6e1

use escape sequence for clearing screen and setting cursor position

3ae4316

Remove unused imports

5bb02a2

Windows repl support

500761e

Arrow key support

98f16b9

Make backspace clear char

911a76a

Fix missing newline after input

77f1c42

Make insert work

243817d

Fix delete in middle

25f51b4

Fix crash on invalid command key

5242239

More fixes

febe424

Remove one more TYPE_CHECKING

d1289af

tonybaloney reviewed May 30, 2024

View reviewed changes

Lib/test/test_pyrepl/test_windows_console.py Outdated Show resolved Hide resolved

Update Lib/test/test_pyrepl/test_windows_console.py

db191a2

Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com>

lysnikolaou added OS-windows topic-repl Related to the interactive shell labels May 30, 2024

zadjii-msft approved these changes May 30, 2024

View reviewed changes

DinoV added 4 commits May 30, 2024 23:10

Use _WindowsConsoleIO for output

fba84aa

Merge branch 'winrepl' of github.com:DinoV/cpython into winrepl

42e3e8e

Remove get_abs_positon

63d854e

Fix tests on windows

513a148

ambv merged commit 0d07182 into python:main May 31, 2024
41 checks passed

bedevere-app bot removed the awaiting core review label May 31, 2024

ambv mentioned this pull request May 31, 2024

gh-111201: [pyrepl] Ensure optional platform-specific imports are optional #119834

Merged

ambv added needs backport to 3.13 bugs and security fixes labels May 31, 2024

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label May 31, 2024

DHowett reviewed May 31, 2024

View reviewed changes

noahbkim pushed a commit to hudson-trading/cpython that referenced this pull request Jul 11, 2024

pythongh-111201: Support pyrepl on Windows (python#119559)

28fa523

Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>

estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024

pythongh-111201: Support pyrepl on Windows (python#119559)

e637683

Co-authored-by: Anthony Shaw <anthony.p.shaw@gmail.com> Co-authored-by: Łukasz Langa <lukasz@langa.pl>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-111201: Support pyrepl on Windows #119559

gh-111201: Support pyrepl on Windows #119559

DinoV commented May 25, 2024 •

edited by github-actions bot

Loading

zadjii-msft left a comment

zadjii-msft May 30, 2024

zadjii-msft May 30, 2024

DinoV May 31, 2024

zadjii-msft May 30, 2024

tonybaloney May 30, 2024

pablogsal commented May 30, 2024

ambv commented May 31, 2024

bedevere-bot commented May 31, 2024

ambv commented May 31, 2024

miss-islington-app bot commented May 31, 2024

miss-islington-app bot commented May 31, 2024

bedevere-app bot commented May 31, 2024

zadjii-msft commented May 31, 2024 •

edited

Loading

DHowett May 31, 2024

ambv May 31, 2024

DHowett May 31, 2024

ambv May 31, 2024


		return Event(evt="key", data=key, raw=rec.Event.KeyEvent.uChar.UnicodeChar)

		def push_char(self, char: int \| bytes) -> None:

gh-111201: Support pyrepl on Windows #119559

gh-111201: Support pyrepl on Windows #119559

Conversation

DinoV commented May 25, 2024 • edited by github-actions bot Loading

zadjii-msft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pablogsal commented May 30, 2024

ambv commented May 31, 2024

bedevere-bot commented May 31, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

ambv commented May 31, 2024

miss-islington-app bot commented May 31, 2024

miss-islington-app bot commented May 31, 2024

bedevere-app bot commented May 31, 2024

zadjii-msft commented May 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DinoV commented May 25, 2024 •

edited by github-actions bot

Loading

zadjii-msft commented May 31, 2024 •

edited

Loading