Make IDLE recognize .pyi stub files as source for open, save, and edit

[AlexWaygood]

BPO	45447
Nosy	@terryjreedy, @ronaldoussoren, @zooba, @miss-islington, @E-Paine, @AlexWaygood
PRs	gh-89610: Add syntax highlighting for .pyi files in IDLE #28950 [3.10] bpo-45447: Add syntax highlighting for .pyi files in IDLE (GH-28950) #31303 bpo-45447: Add entry to What's new 3.10 #31304 bpo-45447: Add entry to What's new 3.9 #31305 [3.9] bpo-45447: Add syntax highlighting for .pyi files in IDLE (GH-28950) #31306 bpo-45447: Fix entry in What's New 3.11 #31307 [3.10] bpo-45447: Add entry to What's new 3.10 (GH-31304) #31308 [3.10] bpo-45447: Add entry to What's new 3.9 (GH-31305) #31309 [3.9] bpo-45447: Add entry to What's new 3.9 (GH-31305) #31310

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2021-10-12.15:24:36.540>
labels = ['expert-IDLE', 'type-feature', '3.11']
title = 'Make IDLE recognize .pyi stub files as source for open, save, and edit'
updated_at = <Date 2022-02-14.03:35:10.254>
user = 'https://github.com/AlexWaygood'

bugs.python.org fields:

activity = <Date 2022-02-14.03:35:10.254>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2021-10-12.15:24:36.540>
creator = 'AlexWaygood'
dependencies = []
files = []
hgrepos = []
issue_num = 45447
keywords = ['patch']
message_count = 28.0
messages = ['403738', '403741', '403755', '403772', '403884', '403889', '403915', '403998', '404021', '404023', '404030', '404100', '404102', '404110', '410637', '410669', '412561', '412617', '413147', '413149', '413150', '413156', '413157', '413158', '413168', '413169', '413170', '413211']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'ronaldoussoren', 'steve.dower', 'miss-islington', 'epaine', 'AlexWaygood']
pr_nums = ['28950', '31303', '31304', '31305', '31306', '31307', '31308', '31309', '31310']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45447'
versions = ['Python 3.11']

Linked PRs

• gh-89610: IDLE: Add util and stub example comments #113222
• [3.12] gh-89610: IDLE: Add util and stub example comments (GH-113222) #113223
• [3.11] gh-89610: IDLE: Add util and stub example comments (GH-113222) #113224

[AlexWaygood]

IDLE currently does not do any syntax highlighting if you use it to open a .pyi stub file, even though everything in a .pyi file is a valid Python expression. It would be great if syntax highlighting for .pyi files could be supported.

[terryjreedy]

It should be fairly easy to recognize x.pyi as a python file. Just have to find the place. Something like
def f(i:int) -> int: ...
would have 'def' and 'int' highlighted.

[AlexWaygood]

It looks like support for .py files is hardcoded into IDLE's source code in the following places:

• browser.ModuleBrowserTreeItem.OnDoubleClick
• browser.ModuleBrowserTreeItem.IsExpandable
• browser.ModuleBrowserTreeItem.listchildren
• editor.ispythonsource << This is the one that controls syntax highlighting
• iomenu.IOBinding.filetypes

If I were to file a PR adding support for .pyi files, would you prefer that I hardcoded ".pyi" in, or would it be better to add a SUPPORTED_FILES constant somewhere?

[terryjreedy]

Thank you for doing the research.

I have been thinking about adding a file containing idlelib 'leaf' objects, those with no idlelib dependencies, which are needed in more than one file. would be to reduce the complexity of the idlelib dependency graph, which has enough cyclic dependencies to make startup tricky. Time to do it. idlelib/common.py?, leaves.py?, or ??? Draft:
---
"""Idlelib objects with no external idlelib dependencies and which are needed in more than one idlelib module. They are included here because a) they don't particularly belong elsewhere or b) because inclusion here simplifies the idlelib dependency graph.

TODO: Python versions (editor and help_about), tk version and patchlevel (pyshell, help_about, maxos?, editor?), std streams (pyshell, run), warning stuff (ditto).
"""

# python_extensions is used in editor, browser, and iomenu.
# .pyw is for Windows; .pyi is for stub files.
python_extensions = ('.py', '.pyw', '.pyi')  
extension_string = "*" + " *".join(python_extensions)

---

editor.EditorWindow.ispythonsource could be moved into the new file if a 'firstline=None' argument were added.

OnDoubleClick should use IsExpandable. IsExpandable should use the logic in ispythonsource so Linux no-extension python source can be browsed as well as edited.

After manual testing, add a new test_common.py file in idlelib.idle_test. After imports, it could start with one test:

class ExtensionTest(TestCase):
    def test_stub(self):
        self.assertIn('.pyi', common.python_extensions)
        self.assertIn('.pyi', common.extension_string)

For IDLE, I can and will backport.

[terryjreedy]

I am going to call the file util.py and will work on a PR.

[AlexWaygood]

Oh — I've already nearly finished a PR. Do you mind if I submit mine? Happy to make any changes you like, obviously.

[AlexWaygood]

Have posted my PR. I've tested it manually and run the IDLE test suite on it with the -u all setting; it seems fine. The new tests I've written are a bit thin on the ground right now, though. I may need some guidance/help if more extensive tests are needed -- I'm a little new to unittest.

One other thing I'm not sure on: the docstring at the top of browser.py includes the line "- finish removing limitation to x.py files (ModuleBrowserTreeItem)" in the TODO items. Can this be removed now?

[E-Paine]

Would it make sense, since this issue pretty much makes IDLE officially support .pyi files, to add the "Edit with IDLE" menu when right-clicking on a .pyi file in Windows explorer?

[terryjreedy]

Yes. After this issue is done, we should open a new issue and nosy Steve Dower, who is in charge of that for Windows, and Ned Deily, in case anything should be done on mac.

[ronaldoussoren]

The most straightforward change for macOS is to add “pyi” to the list of extensions in “Mac/IDLE/IDLE.app/Contents/Info.plist”. That way IDLE will be seen as a valid editor for .pyi files.

[terryjreedy]

Steve, how do we make it so that "Edit with IDLE" appears when right-clicking on a .pyi file in Windows explorer? As easy as on mac (see Ronald's answer above)?

[zooba]

Bit more complicated. It'll need changes to Tools/msi/launcher/launcher_reg.wxs, Tools/msi/tcltk/tcltk_reg.wxs, and PC/layout/support/appxmanifest.py, but I don't think it's that obvious where (we don't want these files to open with py.exe by default, so they'll need a whole new definition).

[AlexWaygood]

Shall I add the change to the mac file to my PR, or would it be better to leave it for a separate PR? (I'm not confident enough to make the changes to the Windows files.)

[terryjreedy]

We can add the mac change once it is manually tested on some system. I may try on my macbook. Windows will be another PR or even issue.

[AlexWaygood]

@terry, have you had a chance to glance at my PR at all? I'd really appreciate your review! :)

[terryjreedy]

A minimal version of util.py just for this issue is fine. What is blocking the issue deciding exactly what the minimum should be. Are both the extension list and function needed?

Alex says somewhere in PR comments that ispythonsource passes directories. That seems wrong. Perhaps moving and revising it and making no-extension files browsable and removing the x.py(?) limitation should be a separate issue. IDLE users on *nix don't mind using extensions. I have not seen a complaint, though perhaps people who do mind silently use something else. I need to check again how the editor function is used and decide if we can leave it alone for now. So don't remove anything just now.

I need to experiment with Ron's suggestion on my Mac.

The Windows context menu is definitely a separate issue.

Please add idle_test/example.pyi with the line in msg403741 for manual testing now and possibly future automatic testing.

[terryjreedy]

For the open dialog, we only need the expanded extension list. People with no-extension python code file must select 'all files'. Once the file is open, we need the issource function to decide whether to turn on the colorizer. I intend to review the patch itself tomorrow.

[terryjreedy]

I broaden the issue title scope and will correspondingly restrict the PR scope. 'Recognition' applies to open and save dialogs, syntax marking, and module browsing. (The latter needs fixing for .pyw also.)

Improving the handling of no extension source with initial #! line will be a different issue.

[terryjreedy]

New changeset 50cf499 by Alex Waygood in branch 'main':
bpo-45447: Add syntax highlighting for .pyi files in IDLE (GH-28950)
50cf499

[terryjreedy]

#89610 (comment) indicates other places that need revision.

[erlend-aasland]

It is not obvious to me what needs revision. Could you elaborate?

[terryjreedy]

All the other places indicated that require python file be .py, thereby excluding .pyi.

[ronaldoussoren]

See also my comment about adding pyi to the list of CFBundleTypeExtensions in IDLE's Info.plist. That will allow opening stub files by double clicking on them in the Finder, and is not necessary for the File/open menu.

Otherwise syntax coloring for stub files appears to work fine in the current Python 3.11 beta.

[erlend-aasland]

See also my comment about adding pyi to the list of CFBundleTypeExtensions in IDLE's Info.plist.

👉🏻 #95393

[erlend-aasland]

It looks like support for .py files is hardcoded into IDLE's source code in the following places:

• browser.ModuleBrowserTreeItem.OnDoubleClick
• browser.ModuleBrowserTreeItem.IsExpandable
• browser.ModuleBrowserTreeItem.listchildren

👉🏻 #95397

• iomenu.IOBinding.filetypes

Seems irrelevant for this issue to me; I might be wrong, I only took a quick glance at the code.

[terryjreedy]

I listed iomenu filetypes before we added the util.py_extensions tuple, which is used in several places. Before, the same information was duplicated in multiple places. That item is now obsolete.

[terryjreedy]

I open task issue #95410 with two of the items completed by this issue. I opened a browser issue #95411 and moved the browser PR there.