Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider not consuming all the buffer in one go in the tokenizer module #105069

Closed
pablogsal opened this issue May 29, 2023 · 1 comment
Closed

Consider not consuming all the buffer in one go in the tokenizer module #105069

pablogsal opened this issue May 29, 2023 · 1 comment

Comments

@pablogsal
Copy link
Member

pablogsal commented May 29, 2023 •
edited by bedevere-bot

Seems that some tools were relying on the implementation detail that the readline-like callable that is provided to functions in the tokenize module is called as tokens are emitted and not consumed in one go. Although this was never part of the contract and technically we don't need to change, it would make the implementation more efficient as we don't need to hold the entire input in memory at the same time and we won't break these tools.

Linked PRs
@pablogsal
Copy link
Member Author

CC: @lysnikolaou @mgmacias95

pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
ff4d45c 
…sume input iteratively
@bedevere-bot bedevere-bot mentioned this issue May 29, 2023
Merged
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
bc6d2da 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
98a180c 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
0a661c4 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
713379f 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
35328a4 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
1b8ff7e 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
pythongh-105069: Add a readline-like callable to the tokenizer to con…
7caac01 
…sume input iteratively

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
pablogsal added a commit to pablogsal/cpython that referenced this issue May 29, 2023
@pablogsal
fixup! pythongh-105069: Add a readline-like callable to the tokenizer…
8903d0d 
… to consume input iteratively
pablogsal added a commit to pablogsal/cpython that referenced this issue May 30, 2023
@pablogsal
fixup! fixup! pythongh-105069: Add a readline-like callable to the to…
d370087 
…kenizer to consume input iteratively
pablogsal added a commit to pablogsal/cpython that referenced this issue May 30, 2023
@pablogsal
fixup! fixup! fixup! pythongh-105069: Add a readline-like callable to…
2d6f0a6 
… the tokenizer to consume input iteratively
pablogsal added a commit to pablogsal/cpython that referenced this issue May 30, 2023
@pablogsal
fixup! fixup! fixup! fixup! pythongh-105069: Add a readline-like call…
0935371 
…able to the tokenizer to consume input iteratively
pablogsal added a commit to pablogsal/cpython that referenced this issue May 30, 2023
@pablogsal
fixup! fixup! fixup! fixup! fixup! pythongh-105069: Add a readline-li…
9990b7e 
…ke callable to the tokenizer to consume input iteratively
pablogsal added a commit to pablogsal/cpython that referenced this issue May 30, 2023
@pablogsal
fixup! fixup! fixup! fixup! fixup! fixup! pythongh-105069: Add a read…
0598127 
…line-like callable to the tokenizer to consume input iteratively
pablogsal added a commit that referenced this issue May 30, 2023
@pablogsal
gh-105069: Add a readline-like callable to the tokenizer to consume i…
9216e69 
…nput iteratively (#105070)
@bedevere-bot bedevere-bot mentioned this issue May 30, 2023
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 30, 2023
@pablogsal @miss-islington
pythongh-105069: Add a readline-like callable to the tokenizer to con…
6de88f2 
…sume input iteratively (pythonGH-105070)

(cherry picked from commit 9216e69)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
pablogsal added a commit that referenced this issue May 31, 2023
@miss-islington @pablogsal
[3.12] gh-105069: Add a readline-like callable to the tokenizer to co…
c687946 
…nsume input iteratively (GH-105070) (#105119)

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (GH-105070)
(cherry picked from commit 9216e69)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
@erlend-aasland erlend-aasland mentioned this issue Jun 6, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pablogsal