[rush] [heft] Allow for config files to have jsonc extension(s) when json files contain comments. (feat proposal)

[jessekrubin]

Summary

Editor setup and syntax highlighting would be nicer and easier (in my opinion) if heft and rush config files were allowed to have jsonc as their file extension.

Several members of my team us webstorm/intellij and a slew of other editors for typescript, many of which hare not as easy to configure as vscode to handle file associations based on paths :/

[iclanton]

That would be useful. We'd have to handle the case when both a .json and .jsonc file exist.

Would you be interested in putting this together?

[octogonz]

That would be useful. We'd have to handle the case when both a .json and .jsonc file exist.

Would you be interested in putting this together?

@iclanton I'm not sure that I agree with this design.

If we allow a mixture of .json and .jsonc then:

1. We double the amount of file I/O required to probe for files in folders.
2. For end users, we create an awkward situation where you have to consider two possibilities every time you need to find a file (bulk search+replace, following a documentation sample, searching discussions for mention of a filename, etc)

If .jsonc is truly superior, then I think we should switch across the board. (Or consider .json5 as discussed earlier in #1088 .)

Perhaps we could:

1. Change the config file extensions across the board, for all Rush Stack tooling. (Maybe this could happen as part of the major version bumps for Rush 6 and Heft 1.0?)

   - OR -

2. Provide a way to configure file extensions across-the-board within a given monorepo. Such a setting would eliminate the performance overhead of probing for different file extensions in every folder, and would at least provide consistent expectations for engineers across projects within one monorepo.

CC @elliot-nelson @chengcyber @dmichon-msft

[dmichon-msft]

We'd probably be better served switching to YAML, since it handles Git merges a lot better. Merge conflicts in JSON files because you modified config settings on adjacent lines get old fast.

[octogonz]

YAML has some downsides:

• The syntax is crazy complicated unless you restrict yourself to a subset.

• It relies on JSON as its paradigm, but some YAML features can't be represented in JSON (e.g. &ref)

• There isn't a normal form, so searching+replacing content isn't very easy for humans, nor is it easy for machines to update a file while preserving its layout.

• Reading/writing YAML requires a 40kb library, which is a pretty heavy dependency for browser apps. By contrast jsonc-parser is only 3k, and there's even cheaper options.

• Seems like YAML might be significantly more expensive to read/write?

Merge conflicts in JSON files because you modified config settings on adjacent lines get old fast.

This problem is particularly annoying in the rush init templates. Whenever you uncomment a setting, you have to add a comma for the previous setting, because Prettier deleted it.

JSON5 allows trailing commas, which I think would solve this, right?

[jessekrubin]

@iclanton @octogonz I could implement it but it is a huge change and think a discussion (such as this one) needs to happen first... I think the default is "if jsonc file exists, use that, but if missing, use json, and throw error if both exist"; but this doesnt really handle the possible doubling of io requests to the file-system... I think the cleanest solution is an "opt-in-use-jsonc-config-setting" and/or a rush/heft package.json config file that says jsonc/json...

@dmichon-msft (what pete said) and my own thoughts:

1. I would absolutely stop being as enthusiastic about rush if a hard switch to yaml were made (I am very pro rush for the record)
2. yaml was designed in medieval Europe as a last-resort torture method/device

[jessekrubin]

This problem is particularly annoying in the rush init templates. Whenever you uncomment a setting, you have to add a comma for the previous setting, because Prettier deleted it.

JSON5 allows trailing commas, which I think would solve this, right?

@octogonz

food for thought: I wrote an auto-installer and global command to use dprint + prettier for src files, but dprint's json/jsonc plugin for json/jsonc files and I haven't had this very problem since

[octogonz]

@iclanton @D4N14L @apostolisms @elliot-nelson

We seem to have gotten quite a lot of feedback about this topic in recent months (via a number of different tangents that start with "I opened a .json file in my editor and all the lines are red!")

Any change to the Rush Stack config files is going to be a nontrivial undertaking, and maybe not our top priority.

HOWEVER that's separate from deciding a position. Are we going to maintain the previous stance, that .json-with-comments is our standard? Should we go down the noncommittal road of config file roulette? Should we plan a bold overhaul?

As a "high" priority: Let's weigh the options and try to reach a consensus, and then we can share that as our official position/goal. That will avoid wasting resources by revisiting this topic endlessly. 😄

[dmichon-msft]

Well, the good news is that as soon as you start searching for 2 files, searching for 20 is the same number of API calls, since at that point you should be querying for a directory listing instead of doing individual existence checks. On the other hand, I am very much in favor of predictable configuration file locations.

For the record, I honestly had no idea YAML had so much other stuff in it (the variable expansion stuff I really didn't expect); I've only ever used it as "expanded comment-friendly merge-friendly JSON", e.g. pnpm-lock.yaml and Azure Pipelines YAML.

I know that the reason we don't like .js for our config files is that at least parsing a JSON file can reasonably be expected to be bounded in time, whereas a JS file could do literally anything. That said, using .js (or .cjs, or .mjs) files for config would solve all the complaints I have about the use of the .json extension for files that aren't syntactically-valid JSON, namely that I want to be able to directly load them with require() or await import(), rather than needing a whole parser library. The fact that right now we have to import an entire parser library just to load a config file is why I see no functional difference between the current state and using YAML.

[jessekrubin]

I thinks worth mentioning that adding $schema to a json file over-rides file associations in many editors.

[octogonz]

I know that the reason we don't like .js for our config files is that at least parsing a JSON file can reasonably be expected to be bounded in time,

Executable code cannot be cached reliably, because we can't be sure that the expressions have no side effects. (For example, consider all the ways that PNPM fails to detect certain kinds of changes from .pnpmfile.cjs.)

A .js config file can't be transformed, for example if we wanted to write an upgrader that reads an old file format and writes a new file format.

It can't be loaded at all by a different programming language. Loading is equivalent to implementing a JavaScript interpreter.

A .js config file can have schema validation, but the error message can't report the line number where the mistake occurred (for example webpack.config.js errors). Doing so would involve tracing program execution to figure out which line of code assigned a field. Even when done by manually using a debugger, this is often not an easy problem.

There are sooo many downsides... :)

[Luxcium]

I do not know why my tsconfig.json is already a JSON with comments out of the box (Opinion: Maybe because the company behind TypeScript or VsCode is awesome)... but I will need to figure out how to fix the issue with RushJS config files like rush.json...

Edit

I have fixed the problem for myself in the workspace .vscode/setting.json config file of my editor:

    {
      // to fix an issue in the RushJS config files 
      "files.associations": {
        "*.json": "jsonc"
      }
    }

[jessekrubin]

@Luxcium that works, but in the intellij ides you cannot set such a file-association and have a $schema key work together :/

[elliot-nelson]

@Luxcium that works, but in the intellij ides you cannot set such a file-association and have a $schema key work together :/

Ah, I don't think I understood the issue before. In IntelliJ, the presence of a $schema forces the document to be a "JSON file matching this schema", and it'll ignore attempts to highlight it as "JSON with comments" instead?

And, it's probably not fixable on the schema side, because a schema can't say a file is "json with comments", that's not a concept that exists in the specification.

[jessekrubin]

@elliot-nelson the problem is fix-able by having a .jsonc extension