Implement pyupgrade

[JonathanPlasse]

• Set literals
• Dictionary comprehensions
• use datetime.UTC alias
• Format Specifiers
• printf-style string formatting
• Unicode literals
• Invalid escape sequences
• is / is not comparison to constant literals
• .encode() to bytes literals
• extraneous parens in print(...)
• unittest deprecated aliases
• super() calls
• "new style" classes
  • rewrites class declaration
  • removes __metaclass__ = type declaration
• forced str("native") literals
• .encode("utf-8")
• # coding: ... comment
• __future__ import removal
• Remove unnecessary py3-compat imports
• import replacements
• rewrite mock imports
• yield => yield from
• Python2 and old Python3.x blocks
• remove six compatibility code
• open alias
• redundant open modes
• OSError aliases
• typing.Text str alias
• Unpacking list comprehensions
• Rewrite xml.etree.cElementTree to xml.etree.ElementTree
• Rewrite type of primitive
• typing.NamedTuple / typing.TypedDict py36+ syntax
  • typing.NamedTuple
  • typing.TypedDict
• f-strings
• subprocess.run: replace universal_newlines with text
• subprocess.run: replace stdout=subprocess.PIPE, stderr=subprocess.PIPE with capture_output=True
• remove parentheses from @functools.lru_cache()
• replace @functools.lru_cache(maxsize=None) with shorthand
• pep 585 typing rewrites
• pep 604 typing rewrites
• remove quoted annotations

Please add the tests from pyupgrade to assure equivalent behavior.

[martinlehoux]

I guess I can just pick some rules in here for implementation?

[JonathanPlasse]

I guess I can just pick some rules in here for implementation?

Yes, can you comment on the one you choose so there is no conflict?

[martinlehoux]

I'll get started on six code

[squiddy]

I'll look at yield -> yield from.

[charliermarsh]

I'd like to knock out a few more of these. Now that the LSP supports autofix, they're even more valuable.

[charliermarsh]

I can do Remove unnecessary py3-compat imports and import replacements.

[colin99d]

I can dig into typing.Text str alias.

[colin99d]

I am going to take these two next:

• subprocess.run: replace universal_newlines with text (UP021)
• subprocess.run: replace stdout=subprocess.PIPE, stderr=subprocess.PIPE with capture_output=True (UP022)

[colin99d]

Today I will work on:

• Rewrite xml.etree.cElementTree to xml.etree.ElementTree (UP023)
• OSError aliases (UP024)
• Unicode literals (UP025)

[colin99d]

Going to start on rewrite mock imports (UP026)

[colin99d]

I will take: import replacements (UP035)

[charliermarsh]

Import replacements might requiring auditing some of what we already do. Like, we do some of the replacements as part of the other Pyupgrade rules. I'm not totally sure what we're missing. (I'm also worried that it'll be a really tricky one to implement 😂)

[colin99d]

I am probably going to regret saying this, but this seems like its not too challenging. The process I was thinking is:

1. Search this dict for matches (converted to a rust hashmap)
2. Replace any specific items that match. If its a muli-line then we can move the replaced items to a new line. I would use libcst imports here
3. Return the new strings

[colin99d]

I will take: Python2 and old Python3.x blocks (UP037)

[colin99d]

I will take: remove quoted annotations (UP038) AKA the LAST one 🎉🎉🎉.

[charliermarsh]

@colin99d - Awesome. Like pyupgrade (IIUC), let's only enforce that when from __future__ import annotations is present.

[Bobronium]

I'm not sure where is a good place to drop this, but it would be great to have pydowngrade (AFAIK it's not a thing yet) along with pyupgrade.

This will allow to develop a project with newest set of features, and maintain compatibility with previous python version by downgrading certain things when building a wheel for specific version.

Hope it's not completely out of place 😄

[colin99d]

Hello Bobronium,

I love the idea! As a developer on a python library myself, it is annoying not being able to add the newest features in my code, because some people are still on python 3.7. However, I see two issues with this idea:

• This would take a while to code (probably just as long as coding all the pyupgrade features), but with a much smaller potential use case.
• Some of the lints would not be able to be reverted, for example, if you ran the lint that removes "u" before a string, then ruff would not know which strings to add it back to.

I think a much better implementation for this would be to have your CI that uploads pip take the version that works for your oldest members, and then upgrade it to the correct version for each user group downloading it. While this doesn't help the developers, it would mean all users of the code get the newest features.

[sjdemartini]

In testing this out using Ruff 0.0.251 and enabling "UP" rules, the first pyupgrade feature mentioned in the original checklist ("Set literals", https://github.com/asottile/pyupgrade#set-literals) does not appear to be handled and is not listed under the rules here https://beta.ruff.rs/docs/rules/#pyupgrade-up, despite being checked off above. Perhaps it was actually never implemented? Should I file a new issue for this?

[charliermarsh]

@sjdemartini - I think those rules are handled instead by flake8-comprehensions rules, like C405.

[sjdemartini]

Aha, indeed, thank you Charlie!

[charliermarsh]

No prob! Same is probably true of dictionary comprehensions.

[aureliojargas]

First of all, thanks for the excellent tool! Loving it 😍

After reading those last comments here, I got curious on exactly which extra rules besides UP I should use to get the full pyupgrade functionality in ruff.

I haven't found that information in the docs, so I did some testing and the closer I got was:

ruff --target-version py311 --select UP,C401,C402,C403,C404,C405,F632,W605 --fix foo.py

Description of those rules:

# Activate all the rules that are pyupgrade-related
select = [
    "UP",    # pyupgrade
    "C401",  # flake8-comprehensions: unnecessary-generator-set
    "C402",  # flake8-comprehensions: unnecessary-generator-dict
    "C403",  # flake8-comprehensions: unnecessary-list-comprehension-set
    "C404",  # flake8-comprehensions: unnecessary-list-comprehension-dict
    "C405",  # flake8-comprehensions: unnecessary-literal-set
    "F632",  # pyflakes: is-literal
    "W605",  # pycodestyle: invalid-escape-sequence
]

The only pyupgrade feature not supported in ruff is remove six compatibility code.

The only ruff UP rule not supported in pyupgrade is UP038: non-pep604-isinstance: Use X | Y in {} call instead of (X, Y).

On a side note, since this PR is highly ranked in the "ruff pyupgrade" search, maybe we could add the ruff rules IDs to the items in this PR description, exactly as being done in "Implement Pylint" (#970)? This way people can better correlate the pyupgrade feature name with the ruff rule. This may help.

[charliermarsh]

Thanks, this is a great comment! I'd love to add the relevant annotations to the issue. If anyone is interested in writing a comment that does that, I'm happy to copy it into the description.

Edit: oh wait, that's basically exactly what you did in your linked issue, thanks @aureliojargas!

[aureliojargas]

Thanks @charliermarsh, I'm happy to provide it:

- [X] Set literals / `C401` `C403` `C405`
- [X] Dictionary comprehensions / `C402` `C404`
- [X] Format Specifiers / `UP030`
- [X] printf-style string formatting / `UP031`
- [X] Unicode literals / `UP025`
- [X] Invalid escape sequences / `W605`
- [X] `is` / `is not` comparison to constant literals / `F632`
- [X] `.encode()` to bytes literals / `UP012`
- [X] extraneous parens in `print(...)` / `UP034`
- [X] unittest deprecated aliases / `UP005`
- [X] `super()` calls / `UP008`
- [X] "new style" classes
    - [X] rewrites class declaration / `UP004`
    - [X] removes `__metaclass__ = type` declaration / `UP001`
- [X] forced `str("native")` literals / `UP018`
- [X] `.encode("utf-8")` / `UP012`
- [X] `# coding: ...` comment / `UP009`
- [X] `__future__` import removal / `UP010`
- [X] Remove unnecessary py3-compat imports / `UP029`
- [X] import replacements / `UP035`
- [X] rewrite `mock` imports / `UP026`
- [X] `yield` => `yield from` / `UP028`
- [X] Python2 and old Python3.x blocks / `UP036`
- [X] remove `six` compatibility code / not ported to ruff
- [X] `open` alias / `UP020`
- [X] redundant `open` modes / `UP015`
- [X] `OSError` aliases / `UP024`
- [X] `typing.Text` str alias / `UP019`
- [X] Unpacking list comprehensions / `UP027`
- [X] Rewrite `xml.etree.cElementTree` to `xml.etree.ElementTree` / `UP023`
- [X] Rewrite `type` of primitive / `UP003`
- [X] `typing.NamedTuple` / `typing.TypedDict` py36+ syntax
    - [X] `typing.NamedTuple`  / `UP014`
    - [X] `typing.TypedDict` / `UP013`
- [X] f-strings / `UP032`
- [X] `subprocess.run`: replace `universal_newlines` with `text` / `UP021`
- [X] `subprocess.run`: replace `stdout=subprocess.PIPE, stderr=subprocess.PIPE` with `capture_output=True` / `UP022`
- [X] remove parentheses from `@functools.lru_cache()` / `UP011`
- [X] replace `@functools.lru_cache(maxsize=None)` with shorthand / `UP033`
- [X] pep 585 typing rewrites / `UP006`
- [X] pep 604 typing rewrites / `UP007`
- [X] remove quoted annotations / `UP037`
- [X] use `datetime.UTC` alias / `UP017`

Please add the tests from pyupgrade to assure equivalent behavior.

BTW, the only change I did was moving the "use datetime.UTC alias" item to the end, because it was the only one not following the order from the pyupgrade README.