Cleaning up tokenize module imports

[podrybaum]

Bug report

Bug description:

This is not really a bug, but I'm preparing to submit a pull request for some minor modifications I made to tokenize.py, to clean up the mess of imports.

Here's what I'm talking about:

from token import *
from token import EXACT_TOKEN_TYPES
...

import token
__all__ = token.__all__ + ["tokenize", "generate_tokens", "detect_encoding",
                           "untokenize", "TokenInfo", "open", "TokenError"]
del token

This is inefficient of course, and a bit hard to follow. As nearly as I can tell, the del token was done to free up the name "token," as it was used in various places throughout the module. I think the name "tok" (which is used in other places all over) is just as descriptive, and doesn't lead to the issues with misinterpreting what's being accessed via the "token" namespace later in the module that can happen if one is quickly scanning through this code.

I'm writing some code that depends on tokenize, so I decided to go ahead and clean it up. Pull request to follow.

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux, Windows

Linked PRs

• gh-128223 - Minor updates to tokenize.py, cleaning up imports and added a comment. #128224

[serhiy-storchaka]

What is inefficient and hard to follow here? This looks normal to me.

token is deleted from globals because it is not used after this line. It does not conflict with local variables names token. This could be written also like:

from token import __all__
__all__ = __all__ + [...]

But such code is less obvious (+= cannot be used here). And since the current code works pretty well, there is no reason to rewrite it.

[picnixz]

(Closing as wont fix as we also decided to close the PR since this would be a purely cosmetic change).