-
-
Notifications
You must be signed in to change notification settings - Fork 29.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-12486: Document tokenize.generate_tokens() as public API #6957
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's also widely used:
https://github.com/search?l=Python&q=generate_tokens&type=Code
The test failures appear to be unrelated to this pull request (there's a |
Close/reopen, hopefully that will rerun tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Thanks @takluyver
@willingc: Please replace |
Thanks ! |
I think discussions of tokenizing unicode have often got caught up in trying to design nicer APIs for it. I am taking the much lazier approach of promoting the existing but undocumented
generate_tokens
API to a public API.Why should we do this?
generate_tokens()
has been there for several releases already (at least since 3.3.0), so we can start using it without having to wait until a new Python version is widespread.tokenize()
, so it's easy to understand.https://bugs.python.org/issue12486