Prettier Compatibility Metrics and Goals

[NicholasLYang]

We've set a goal for ourselves of being "Prettier compatible", i.e. our formatter should produce an output very similar to Prettier. This is a good goal for a couple reasons: It punts any larger discussion on formatting issues and it makes adoption easier by preventing large git diffs. Not to mention, Prettier's formatting is overall quite nice.

As nice of a goal as this is, it's hard to define what "Prettier compatibility" actually means. The most optimistic goal would be a precise 1 to 1 match for Prettier output for all JavaScript/TypeScript code. That's not really feasible. It'd take too long and it'd require a lot of porting of Prettier's logic to our codebase. Prettier also has some very very specific edge cases which we may not want to support.

Therefore, what should our goal be? We should create some form of metric for Prettier compatibility. I'm not sure what that metric will be, but I'll note down some potential pitfalls that we should avoid:

• Overfitting: currently we just test against the tests that Prettier provides and some custom tests of our own. Therefore we accept code that may only be compatible with Prettier for these specific tests. Often times we don't even test against Prettier, instead relying on eye tests with the playground.
• Niche differences: Prettier has a lot of rules. Realistically we're not going to implement all of them right now. We should avoid a metric that punishes differences that don't show up in code in practice
• Perfect being the enemy of good: We should not spend too much time on this metric. While Prettier compatibility is important, we cannot spend months chasing down this tail. We should do our best and accept that we will fall short and have to ship at some point.

[ematipico]

You raised a really valid point Nicholas. We should really draw a big line and say "We are satisfied with the level of compatibility of the code (how equal they are)". And that's it. Once we draw the line, rome formatter will start being the actual rome formatter and we will take our decisions based on our judgement.

The great and big problem is that Prettier, as software, will never reach the "feature complete" status (the languages will continue to evolve, as much as the software itself), which means that we will need to have at least a couple of people to follow the project, understand their code, porting code, etc. which is really time consuming and I don't think it's our goal.

I assume that our goal is to make the users comfortable with the transition, by avoiding too many changes. Which means that changes will still occur due to different reasons:

• formatter architecture
• tech stack
• available time of the team
• amount of work for new features/fixes

[NicholasLYang]

I agree, it’s probably not feasible to track prettier extremely closely. What I’m thinking we could do is start reaching out to open source projects, helping them migrate to Rome (open a PR that switches prettier to Rome), see how much of a diff that is, and making the according changes. We could even do this internally but I do think getting feedback from actual users would be helpful. I wouldn’t be surprised if for some small projects they’d be down already.

I think any other metric might end up overfitting. We should fit for user acquisition

[NicholasLYang]

By internally I mean we could take a popular open source project, run Rome, see the diff, and determine if that’d be a reasonable PR that’d pass review

[NicholasLYang]

I'd like to propose a specific level of compatibility: Pull Request Compatibility. What this means is that if a PR were to be opened that switches a project from Prettier to Rome, the amount of changes would be sufficiently small that people would be willing to merge the PR. Of course, this is a subjective measurement, but we can start with small one-person projects and then scale up to larger projects.

[ematipico]

How do we measure "sufficiently small"? Should we draw some kind of proportion between LOC and the number of changes?

[NicholasLYang]

Yeah like a ratio of LOC changed to total LOC? That could work. I think it's also important to ask the maintainer. It could be that some people are fine with big diffs or some people are absolutely intolerant of large changes. We can only find out by asking.

[MichaReiser]

These is roughly the metric that we have today. It's defined as

$$matching/MAX(lines(rome), lines(prettier))$$

What I think would be a good idea is if we explore a setup similar to xtask_bench where we apply the formatter to various open source projects (rome, babel, some vue project, some reacht project, a TS project) and then compare the overall similarity to get a feeling how close we are in real-world use cases.

[NicholasLYang]

As an example, I ran Rome on wasup, my library for generating WebAssembly: https://github.com/NicholasLYang/wasup/pull/1/files

If I were to see a PR like this, I'd be a little concerned due to the large amount of changes. That said, it does seem like a lot of these changes can be eliminated by fixing trailing commas.

[NicholasLYang]

Another topic worth discussing is our choice of formatting options. I'd like to propose that we explicitly follow Prettier's default options as our default options. That means we avoid trailing commas in situations other than objects and arrays, and we indent with two spaces.

As for adopting other formatting options, while I'm aware that the ideology of the formatter is to keep a minimal set of options, we should acknowledge that missing options will limit adoption. People who explicitly configure prettier probably care about these options and won't want to move to a formatter where they can't use them. There might be possible exceptions for people who adopted code, whether from a starter pack or an existing codebase, that had these options and don't care themselves about these minutiae.

[NicholasLYang]

I did a quick search in GitHub of .prettierrc files with different options. I got the following results:

singleQuote: 1,143,491 results
trailingComma: 913,527 results
semi: 686,468 results
printWidth: 574,086 results
tabWidth: 571,112 results
arrowParens: 473,514 results
bracketSpacing: 401,413 results
useTabs: 337,924 results
jsxSingleQuote: 83,743 results
quoteProps: 62,885 results
bracketSameLine: 23,129 results

[MichaReiser]

Would you mind sharing the queries you used to get these numbers?

I don't think we should change our defaults just because of that. The idea is to have a cli command that automatically configures our formatter for you. That command can determine the defaults by:

• inspecting the source files and determining the default indent and new line characters
• Migrating any options from prettier or editorconfig

This should reduce the churn but allow us to use our own defaults.

I agree that missing options limits adoption but see the previous discussion what the reasoning is behind the option and defaults we picked. I don't mind re-visiting some of these decisions but don't think it's worth re-starting from scratch.

Also, not supporting some of these options doesn't mean that we won't support them ever. But that we de-prioritised them for now, considering that there are many more pressing issues to solve first.

[NicholasLYang]

I think a migration tool could be very helpful, of course. However, that means another tool to build. If we could make it easier for people to do it manually with less effort on our part, that might be worth exploring. I know that my first impression upon running Rome on a codebase that uses Prettier was that there were a bunch of changes. When I tweaked the settings it went down, but I’d prefer to not have that initial impression. It’d make sense to have our options default to Prettier’s default if we’re pushing prettier compatibility.

[MichaReiser]

The challenge with this is that it also means that we have to support all options that prettier does which requires time too.

Investing into a migration tool is something that isn't just useful for a prettier to rome migration but also later when setting up a new project. That's why I believe that investing some time into that (with persistent options, a feature we need anyway), provides more value in the long term.

If you disagree, please suggest specific options fir which we should change the default and if this is just a change of default but also requires adding support fir this option in the first place