IDLE shell uses wrong namespace for completions

[taleinat]

BPO	37821
Nosy	@terryjreedy, @taleinat
PRs	gh-82002: IDLE completion namespace fix #15207

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2019-08-11.11:17:08.862>
labels = ['3.8', 'expert-IDLE', 'type-bug', '3.7', '3.9']
title = 'IDLE shell uses wrong namespace for completions'
updated_at = <Date 2019-08-14.03:02:56.017>
user = 'https://github.com/taleinat'

bugs.python.org fields:

activity = <Date 2019-08-14.03:02:56.017>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2019-08-11.11:17:08.862>
creator = 'taleinat'
dependencies = []
files = []
hgrepos = []
issue_num = 37821
keywords = ['patch']
message_count = 5.0
messages = ['349386', '349387', '349437', '349557', '349634']
nosy_count = 2.0
nosy_names = ['terry.reedy', 'taleinat']
pr_nums = ['15207']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37821'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[taleinat]

Currently, when running an IDLE shell with a sub-process, it will allow completing attributes of objects not actually in the shell's namespace. For example, typing "codecs." will bring up completions for the codecs module's attributes, despite that not having being imported. Further, if one uses the completion, this results in a NameError exception, since "codecs" isn't actually in the namespace.

AFAICT, the intended behavior is as follows:

• If a shell exists, completion should use the namespace used for evaluating code in the shell. (Note that this is slightly different when running a shell without a sub-process.)
• If no shell exists (only editor windows are open), completion should use the global namespace + sys.modules.

[taleinat]

See fix in #15207.

[terryjreedy]

I think the current behavior is intended. The IDLE doc Completions subsection, written by Kurt Kaiser in help.text and copied to idle.rst in bpo-5066, has this paragraph

Completions are currently limited to those in the namespaces. Names in an Editor window which are not via __main__ and sys.modules will not be found. Run the module once with your imports to correct this situation. Note that IDLE itself places quite a few modules in sys.modules, so much can be found by default, e.g. the re module."

The second sentence is garbled but says that things in sys.modules *will* be found. The fourth sentence points out that sys.modules has many things imported by IDLE, not by user code, so they can be found *without running one's code*. The example is "import re...re." and re completions pop up. I noted on bpo-18766 that cleaning up run.py made this *feature* less useful. It appears that this issue and your patch are about disable this intended feature. I don't agree.

To put it another way, "If a shell exists, completion should use the namespace used for evaluating code in the shell." is not correct. The intended behavior is to complete if at all possible. bpo-18766 is about expanding 'possible'.

To constantly maximize 'possible', one would run after every name-binding statement (an extreme version of the 3rd sentence above). I sometimes come close to that, though more to catch typos and rerun tests as I go.

You are correct that entering 're.DOTALL' by completion without also entering the required corresponding import results in a NameError when running the code . But the same is true if one enters the same 're.DOTALL' by key. Either way, one has entered the attribute where and when wanted; the NameError says "now go back up and add the import".

What is odd to me is that you seem happy using the more numerous non-user IDLE imports when there is no Shell. That would mean that users would initially be better off killing the shell when starting to edit. To me, the logic of restricting completions when there is a shell would mean no completions when there is not.

[taleinat]

Terry, many thanks for the clarifications!

The intended behavior is to complete if at all possible.

I can see the point in that.

What is odd to me is that you seem happy using the more numerous non-user IDLE imports when there is no Shell.

The alternative would just be completing in an empty namespace, which would be completely useless. I do agree that having a shell should increase the available modules etc. available for completion.

FWIW I brought this up trying the dict-key completions in the shell with globals()[ and got surprising results.

Perhaps we should differentiate between completions in the shell vs. when editing a file. In the shell, having completions for things not actually available is weird IMO. In the editor it makes more sense.

Or perhaps we should really go the route of issue bpo-18766 and try to make all available modules available for completions. I really dislike the arbitrary availability of only certain modules for completion. That would likely be a *huge* undertaking, though, and I don't think it's worth it. Maybe restrict this to just stdlib modules, and/or use lazy (on demand) importing.

Finally, we can also decide that the current behavior is what we want. It's there, documented, and normally not very surprising.

[terryjreedy]

The current behavior is the quick and easy/dirty solution that we have, not what we especially want. bpo-27609 has multiple issues and ideas for completion improvements. I made testing and improving the code priorities before worrying about most of the proposed changes.

Here is where we got confused a bit. The title refers to shell completions. The opening post starts with shell completions and then moves to intended behavior for editor completions and its dependence on the presence of a shell. I disagreed with the latter and responded, not thinking much about shell completions. Lets stick to shell completions, at least initially (and change the title if we include editor completions).

Tal: "Perhaps we should differentiate between completions in the shell vs. when editing a file." I agree that we can consider adjusting the list according to the setting. It is easy enough. Open_completions, executed in the idle process, can detect whether the associated text is the shell or an editor and pass a (new) flag to fetch_completions, which will pass it to the fetch_completions executed in the run process.

Tal: "In the shell, having completions for things not actually available is weird". That I can see and agree to. While the editor allows after-completion revisions anywhere in the file before running, the user cannot revise Shell history and insert an executed import before executing the current statement. A doc statement like "Name and attribute completion in the shell is restricted to names and objects available after previous shell statements." seems clear enough to anyone who reads it.

As near as I can tell, both before and after the bpo-21261 patch, name (tab) completion *is* restricted to valid names. But attribute completion will complete for modules such as socket import by run.py even when not in the main namespace. Python in masOS Terminal (bash, I believe) does attribute completion (.<tab>), but only for user visible names, not, for instance, for reprlib. The proposal is to restrict IDLE's Shell's attribute completion similarly.

Tal: "I really dislike the arbitrary availability of only certain modules for completion." It is not completely arbitrary. We could complete stdlib names from a pre-generated list. But that would add perhaps 400 names to the name completion list. But we can only complete attributes (which I, at least, use much more) for modules in sys.modules. A possible improvement mentioned more than once is on-demand import, at least of stdlib modules. If any are shown to be a problem, we could make a blacklist.