Tokenizer/untokenizer seems to break when linting multiline strings with variables/emojis

[hemanthkini]

Bug report

Bug description:

Linting this file with the latest versions of pyupgrade and docformatter on Python 3.12.4:

def test(test_arg1):
    content = f"""
    <table style="width: 100%; height: 100%; font-size: 1.1em;">
        <tr>
            <td style="width: 100%; height: 100%;">🔗 This is a test {test_arg1} </td>
        </tr>
    </table>
    """
    return content

seems to break:

def test(test_arg1):
    content = f"""
    <table style="width: 100%; height: 100%; font-size: 1.1em;">
        <tr>
            <td style="width: 100%; height: 100%;">🔗 This is a test {te{test_arg1} </td>
        </tr>
    </table>
    """
    return content

I filed PyCQA/docformatter#282 and asottile/pyupgrade#949; the maintainer of pyupgrade directed me here. I wasn't able to find a duplicate f-string or tokenizer issue, so hoping this is of help - thanks!

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

[pygeek]

Try upgrading to 3.12.3 or later. It's possible that this bug was fixed.
See: #115171 .

[asottile]

this is a regression in 3.12.4 -- here's the difference in tokenization output --

--- 3_12_3_output	2024-06-11 23:49:06.250595428 -0400
+++ 3_12_4_output	2024-06-11 23:48:09.954459741 -0400
@@ -11,10 +11,10 @@
 2,12-2,13:          OP             '='            
 2,14-2,18:          FSTRING_START  'f"""'         
 2,18-5,68:          FSTRING_MIDDLE '\n    <table style="width: 100%; height: 100%; font-size: 1.1em;">\n        <tr>\n            <td style="width: 100%; height: 100%;">🔗 This is a test '
-5,68-5,69:          OP             '{'            
-5,69-5,78:          NAME           'test_arg1'    
-5,78-5,79:          OP             '}'            
-5,79-8,4:           FSTRING_MIDDLE ' </td>\n        </tr>\n    </table>\n    '
+5,71-5,72:          OP             '{'            
+5,72-5,81:          NAME           'test_arg1'    
+5,81-5,82:          OP             '}'            
+5,82-8,4:           FSTRING_MIDDLE ' </td>\n        </tr>\n    </table>\n    '
 8,4-8,7:            FSTRING_END    '"""'          
 8,7-8,8:            NEWLINE        '\n'           
 9,4-9,10:           NAME           'return'

biisected to 4a0af0c

cc @lysnikolaou @pablogsal

[lysnikolaou]

Thanks for the report!

Duplicate of #120343.