Implement autoformat capabilities

[charliermarsh]

This issue is the public kickoff for Ruff's autoformatting effort.

A few comments on how I'm thinking about the desired end-state:

• The goal is to enable users to replace Black with Ruff. So, ideally, new projects could come in and replace pyflakes, pycodestyle, isort, and black with a single tool (Ruff).
• As with the rest of Ruff, autoformatting will be entirely optional. You can continue to use Ruff alongside Black. You can also use Ruff just as an autoformatter.
• I'd like to adhere to Black's formatting choices as much as possible, but exact 1:1 parity with Black is not a goal. I'm comfortable deviating when we can make a compelling case, but I'd like that case to be based on arguments that are unrelated to code style itself (e.g., simpler implementation).
• I'd like to enable slightly more configuration than is possible with Black (see: Replacing black and blue? #813), using Prettier as a model. Prettier allows you to specify the line-length, tab width, quote style, trailing comma behavior, and one or two other stylistic features. I think it's reasonable to support at least line length, indentation, and quote style, so I plan to start there. (It's not a goal to be as flexible as, say, Rustfmt.)

With big projects like this, I work best by taking some time to hack on possible implementations, so I'd like to do some free exploration before opening up the issue to other contributors. But I'd love to hear feedback on the above (and any tips you may have) as I start working on this.

[ichard26]

While I'm not sure how to feel about seeing Black being replaced as one of its maintainers (I'm kinda attached TBH!) I'd be happy to answer any questions you might have about Black. Feel free to ask about its code style and its implementation.

I barely know anything about Ruff, but from what I'm hearing, it's probably going to be the future of code quality tooling. There's no way we (Black) can ever compete on speed or being an all-in-one solution. I'm not going to contribute code or whatever, but I do feel like I have a responsibility to our users to build a better formatter even if that means helping arguably a competitor.

Obviously I do have my own Black-influenced opinions on "good code style" so don't expect me to suggest significant deviations from Black's style. I couldn't do that to my project, that's too much of a betrayal :)

[charliermarsh]

Thank you for the thoughtful message, and for the kind offer -- I really appreciate it.

To be honest, I was a little hesitant to post this Issue, since I too am a fan and long-time user of Black, and I don't want to send the wrong message by offering an alternative. But, it's just a very natural extension of what Ruff is already doing, and it's something that comes up constantly when talking to users.

(Also: while I do want Ruff to be a viable all-in-one solution, I also want it to be incrementally useable and incrementally adoptable -- so, e.g., even if we were to complete this, my intention is that it'd always be possible to use Black alongside Ruff. Ruff has the same relationship with isort right now.)

I'll keep your offer in mind as I get the ball rolling here :)

[LefterisJP]

Hey @charliermarsh!

Can I chime in and ask for maybe something different? Black is really good at what it's doing and it's also fast (enough). What I see as a really good way for ruff to go for auto-formatting would be the way of yapf .... but better. So faster and more customizable. Black exists and we don't really need another black.

What python lacks is a customizable auto-formatter that can be customized by each project to come as close to their preferred style as possile. Only thing close to that is yapf.

yapf is an auto-formatter tool for python based on the ideas behind clang format. So the idea is to make it as customizable as possible for projects to apply their rules via autoformat.

The current problems with yapf are:

1. It's not as customizable as it can be.
2. It's super slow for big projects. Takes 4-5 minutes for rotki.
3. I think it's not in a very active maintenance mode

I really believe ruff can shine here and would love to see it become this kind of fast and customizable auto-format tool.

[warsaw]

Great to see your thoughts here, and I'll watch closely with a hope of contributing if I have the bandwidth, and at least testing things out when you have stuff to try.

Some thoughts:

• Quote style is probably the number 1 reason why blue even exists. It's not good enough to just not change quote style; blue deliberately chooses single quotes over double quotes wherever possible, for reasons we think are good, but in keeping with the spirit of this ticket, I won't argue that right now.
• We think blue does a better job of maintaining spacing between code and any right hanging comments, although not a perfect job.
• Blue is really difficult to maintain because it monkeypatches black (to gain from all the really great work that black and its maintainers have done!). Monkeypatching of course is extremely fragile. Without proper APIs in black (and I don't fault black for not supporting them), blue will always have a difficult time keeping up.
• Blue has (had?) better configuration options than black, but even there it's problematic. Having to keep up with the internal implementation details of two external projects now is pretty daunting.
• I personally don't care about the wide features of YAPF. For me and my projects (both open source and for $work), black gets us almost all the way there. I remember sending Łukasz a list of like 10-11 bullet points about black's choices back in the early days, and he refuted every one of them. Which of course is perfectly okay! 😄 - in fact, I've accepted and/or come around on all of them but the few that I still feel blue makes better choices on.

All that to say that IMO, it's worth having some choice on formatting styles, blue tries to make that possible, but maintaining it is difficult, the black developers are doing a fantastic job, and I still would like to see ruff provide a good, fast alternative!

[layday]

@warsaw

Quote style is probably the number 1 reason why blue even exists. It's not good enough to just not change quote style; blue deliberately chooses single quotes over double quotes wherever possible, for reasons we think are good, but in keeping with the spirit of this ticket, I won't argue that right now.

Just wanna point out that you can accomplish this right now with black -S + ruff --fix and the appropriate [tool.ruff.flake8-quotes] config - it's what I use :)

(Will collapse this since it's OT.)

[WhyNotHugo]

There's one thing that black won't do and I think ruff would be able to address: E501. When a line is too long, black will often ignore that. For example:

hello = "This is a really long string that's really over 80 characters wide, and black won't do anything about it"

It would be kinda weird if ruff doesn't autoformat this and complains that it's over 88 characters long. The solution to the above should be:

hello = (
    "This is a really long string that's really over 80 characters wide, and "
    "black won't do anything about it"
)

[charliermarsh]

@WhyNotHugo - Black can fix that if you use black --preview :) Hopefully Ruff can support this too.

[timabbott]

This project is very exciting for Zulip! To explain some of the practical benefit, one of the main practical limitations the size of a Python file is that Black takes about a 200ms per 1K line of code, which becomes a very noticeable lag when running that as an on-save hook in an editor. If that part was at Ruff speeds, I'd expect there to be essentially no autoformatter related lag when saving files in an editor, which would be a visible improvement to our Python development experience.

So being able to replace Black with Ruff, as we have with most of our other Python linters would be amazing. I support the design decision to support slightly more configurability than Black has -- while I don't recall the details, I remember our migration to Black being delayed for a long time because they didn't have the options to avoid some things that we considered style regressions.

[charliermarsh]

@timabbott - That's great to hear and I appreciate you chiming in :)

[charliermarsh]

Brief update: I'm continuing to work on this :)

It's going well! Several tests from the Black suite are passing, more are close-to-passing (I'd like to quantify this soon, but I haven't done string normalization yet, and that causes most tests to fail even if the rest of the formatting is correct).

Still a lot of work and cases left to handle. But my current thinking is that I'll OSS it as soon as I'm confident that it can handle all syntax including comment preservation, even if the style drifts a bit over a few releases as we improve compatibility.