urlquote / f-string issue

[spookylukey]

Python Version

3.8

Django Version

3.2.9

Package Version

1.4

Description

django-upgrade changed this import:

-from django.utils.http import urlquote
+from urllib.parse import quote

But then failed to change usages of urlquote to quote. The only usage in my case was inside an f-string, don't know if that is relevant.

I'll put more effort into a reproducible test case if needed

[adamchainz]

Oh fun.

Turns out the tokenize leaves f-strings as a single token:

In [1]: import tokenize_rt

In [2]: tokenize_rt.src_to_tokens('f"{x()}"')
Out[2]:
[Token(name='STRING', src='f"{x()}"', line=1, utf8_byte_offset=0),
 Token(name='NEWLINE', src='', line=1, utf8_byte_offset=8),
 Token(name='ENDMARKER', src='', line=2, utf8_byte_offset=0)]

Whilst the AST parses them, presumably through a second internal tokenization step:

In [3]: import ast

In [4]: print(ast.dump(ast.parse('f"{x()}"'), indent=2))
Module(
  body=[
    Expr(
      value=JoinedStr(
        values=[
          FormattedValue(
            value=Call(
              func=Name(id='x', ctx=Load()),
              args=[],
              keywords=[]),
            conversion=-1)]))],
  type_ignores=[])

The fixer finds the AST Name for the old name, but its callback is then never called because there's no token corresponding to the Name.

I'm not sure what we can do here... perhaps recursively apply the fixers on FormattedValues. Maybe there's something in pyupgrade to copy.

[adamchainz]

The pyupgrade issue ( asottile/pyupgrade#572 ) was just resolved because it "just works" on Python 3.12. This is thanks to the PEP that modified how f-strings are parsed. I will add a 3.12+ test here to check this is also the case for django-upgrade.