gh-105017: Include CRLF lines in strings and column numbers #105030

mgmacias95 · 2023-05-27T21:46:42Z

Issue: Tokenizer produces different output on Windows on py312 for ends of files #105017

Python/Python-tokenize.c

pablogsal · 2023-05-27T23:11:12Z

There were a bunch of failing tests because we are removing CRLF handling in the regular path that expects the source to be normalized. I pushed a commit that only preserves CRLF in our tokenize extension

Lib/test/test_tokenize.py

bedevere-bot · 2023-05-28T12:50:37Z

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 363de28 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

bedevere-bot · 2023-05-28T13:40:11Z

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit d3d4ff5 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

miss-islington · 2023-05-28T14:15:57Z

Thanks @mgmacias95 for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

…thonGH-105030) (cherry picked from commit 96fff35) Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

bedevere-bot · 2023-05-28T14:16:08Z

GH-105041 is a backport of this pull request to the 3.12 branch.

…H-105030) (#105041) gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (cherry picked from commit 96fff35) Co-authored-by: Marta Gómez Macías <mgmacias@google.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

pablogsal

LGTM

Great job! ✨

bedevere-bot · 2023-05-28T15:40:03Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.12 has failed when building commit 3f8d5d9.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1102/builds/33) and take a look at the build logs.
Check if the failure is related to this commit (3f8d5d9) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1102/builds/33

Failed tests:

test_threading
test__xxsubinterpreters

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 16, done.        
remote: Counting objects:   6% (1/16)        
remote: Counting objects:  12% (2/16)        
remote: Counting objects:  18% (3/16)        
remote: Counting objects:  25% (4/16)        
remote: Counting objects:  31% (5/16)        
remote: Counting objects:  37% (6/16)        
remote: Counting objects:  43% (7/16)        
remote: Counting objects:  50% (8/16)        
remote: Counting objects:  56% (9/16)        
remote: Counting objects:  62% (10/16)        
remote: Counting objects:  68% (11/16)        
remote: Counting objects:  75% (12/16)        
remote: Counting objects:  81% (13/16)        
remote: Counting objects:  87% (14/16)        
remote: Counting objects:  93% (15/16)        
remote: Counting objects: 100% (16/16)        
remote: Counting objects: 100% (16/16), done.        
remote: Compressing objects:   6% (1/15)        
remote: Compressing objects:  13% (2/15)        
remote: Compressing objects:  20% (3/15)        
remote: Compressing objects:  26% (4/15)        
remote: Compressing objects:  33% (5/15)        
remote: Compressing objects:  40% (6/15)        
remote: Compressing objects:  46% (7/15)        
remote: Compressing objects:  53% (8/15)        
remote: Compressing objects:  60% (9/15)        
remote: Compressing objects:  66% (10/15)        
remote: Compressing objects:  73% (11/15)        
remote: Compressing objects:  80% (12/15)        
remote: Compressing objects:  86% (13/15)        
remote: Compressing objects:  93% (14/15)        
remote: Compressing objects: 100% (15/15)        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 16 (delta 1), reused 11 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  3.12       -> FETCH_HEAD
Note: switching to '3f8d5d9ed60ae0f70c92dc29587cc6367eda80c3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3f8d5d9ed6 [3.12] gh-105017: Include CRLF lines in strings and column numbers (GH-105030) (#105041)
Switched to and reset branch '3.12'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: Segmentation fault

Thread 0x00007f253095a280 (most recent call first):
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 74 in _write_status
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 95 in addSuccess
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/testresult.py", line 122 in addSuccess
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 648 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/case.py", line 690 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 122 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/suite.py", line 84 in __call__
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/unittest/runner.py", line 240 in run
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1115 in _run_suite
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/support/__init__.py", line 1241 in run_unittest
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 294 in _test_module
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 330 in _runtest_inner2
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 373 in _runtest_inner
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 248 in _runtest
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/runtest.py", line 278 in runtest
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 365 in rerun_failed_tests
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 783 in _main
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 738 in main
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/libregrtest/main.py", line 802 in main
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/test/__main__.py", line 2 in <module>
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 88 in _run_code
  File "/root/buildarea/3.12.angelico-debian-amd64/build/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi (total: 1)
Segmentation fault
make: *** [Makefile:2009: buildbottest] Error 139

Cannot open file '/root/buildarea/3.12.angelico-debian-amd64/build/test-results.xml' for upload

fix(tokenizer): Include CRLF lines in strings and column numbers

ad130da

mgmacias95 requested review from pablogsal and lysnikolaou as code owners May 27, 2023 21:46

bedevere-bot added the awaiting review label May 27, 2023

bedevere-bot mentioned this pull request May 27, 2023

Tokenizer produces different output on Windows on py312 for ends of files #105017

Closed

mgmacias95 and others added 2 commits May 27, 2023 22:48

Merge branch 'main' into crlf_strings

6688c36

📜🤖 Added by blurb_it.

e5293bc

pablogsal added the needs backport to 3.12 bug and security fixes label May 27, 2023

Only preserve crlf in the Python-tokenize extension

37f77ad

pablogsal reviewed May 27, 2023

View reviewed changes

Python/Python-tokenize.c Show resolved Hide resolved

pablogsal changed the title ~~gh-105017: [Tokenizer] Include CRLF lines in strings and column numbers~~ gh-105017: Include CRLF lines in strings and column numbers May 27, 2023

pablogsal added 4 commits May 28, 2023 12:32

Handle \r\n in continuation lines

68e7b36

fixup! Handle \r\n in continuation lines

969060f

fixup! fixup! Handle \r\n in continuation lines

559487c

fixup! fixup! fixup! Handle \r\n in continuation lines

363de28

pablogsal reviewed May 28, 2023

View reviewed changes

Lib/test/test_tokenize.py Outdated Show resolved Hide resolved

pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

mgmacias95 and others added 2 commits May 28, 2023 14:55

Update Lib/test/test_tokenize.py

67b3a9c

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>

Prevent leak

d3d4ff5

pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 28, 2023

pablogsal merged commit 96fff35 into python:main May 28, 2023
36 of 43 checks passed

bedevere-bot removed the awaiting review label May 28, 2023

bedevere-bot removed the needs backport to 3.12 bug and security fixes label May 28, 2023

pablogsal reviewed May 28, 2023

View reviewed changes

picnixz mentioned this pull request May 28, 2023

Compatibility: pycode parser: adjust dedent-token handling to maintain py3.12+ compatibility sphinx-doc/sphinx#11440

Closed

erlend-aasland mentioned this pull request Jun 6, 2023

3.12 backport gh 105236 #105358

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-105017: Include CRLF lines in strings and column numbers #105030

gh-105017: Include CRLF lines in strings and column numbers #105030

mgmacias95 commented May 27, 2023 •

edited by bedevere-bot

pablogsal commented May 27, 2023

bedevere-bot commented May 28, 2023

bedevere-bot commented May 28, 2023

miss-islington commented May 28, 2023

bedevere-bot commented May 28, 2023

pablogsal left a comment

bedevere-bot commented May 28, 2023

gh-105017: Include CRLF lines in strings and column numbers #105030

gh-105017: Include CRLF lines in strings and column numbers #105030

Conversation

mgmacias95 commented May 27, 2023 • edited by bedevere-bot

pablogsal commented May 27, 2023

bedevere-bot commented May 28, 2023

bedevere-bot commented May 28, 2023

miss-islington commented May 28, 2023

bedevere-bot commented May 28, 2023

pablogsal left a comment

Choose a reason for hiding this comment

bedevere-bot commented May 28, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

mgmacias95 commented May 27, 2023 •

edited by bedevere-bot