Request: lex JSON even if it contains comments

[JulianGro]

JSON code blocks containing comments cannot be lexed. While the standard does not allow comments, it makes sense to diverge from this limitation in some case. One of these cases is documentation. In my case it is also comments commonly being used in JSON files on the Vircadia platform. Without these comments end users won't be able to understand what a field does.

Ideally pygments would lex JSON code blocks even if they contain comments.

In my case we use // to mark comments, same as on JavaScript.
Here is an example:

{
  "useConfirmDialog": true,
  "confirmDialogMessage": "Are you sure you want to open this link?",
  "url": "https://docs.vircadia.dev/",
  // options are "interface" and "browser"
  // "interface" opens an overlay, "browser" opens the OS' external browser.
  "openIn": "interface",
  "dimensions": {
    "width": 1600,
    "height": 900
  }
}

[Mesteery]

#1680 (comment)

[JulianGro]

Problem for me is that many people don't know that comments aren't a part of JSON. Even some manuals like https://www.w3schools.com/js/js_json_intro.asp use comments without saying anything about them not being allowed. (They use comments in their JSONs all over the place)

But yeah, we use pygments as part of Sphinx and set up a test that fails if there is any warnings (such as a JSON not being lexed) so I our case it would just be an inconvenience in the sense that I would constantly have to explain to people why what they think is JSON isn't being lexed by the JSON lexer.

[Mesteery]

That doesn't surprise me coming from w3school because it's a bad source (bad practices, misinformations, etc.).

But yeah, we use pygments as part of Sphinx and set up a test that fails if there is any warnings (such as a JSON not being lexed) so I our case it would just be an inconvenience in the sense that I would constantly have to explain to people why what they think is JSON isn't being lexed by the JSON lexer.

You can put a warning.

I think it's no longer a problem with Pygments at this stage,

[Anteru]

Closing this -- not sure what else to say beyond "your JSON is not JSON" if it has comments.

[kurtmckee]

I've added comment support to JSON in #2049. It's not part of the JSON spec but adding it doesn't affect parsing of valid JSON documents.

[Marrin]

IIRC JSON had comments initially and Crockford immediately saw them misused to put extra information into them, destroying the idea of a simple data interchange format that can't be disrupted by (de)serialisers that rely on extra meta information in comments. So he decided to kick comments out of the JSON spec.

Now you add comments to a JSON lexer — why? It's not valid JSON any more with comments. And if you want to highlight this, well JavaScript, there's a JavaScript lexer already there.

[kurtmckee]

It's not valid JSON any more with comments.

Yes, so don't include comments in your JSON files and they'll continue to be valid. Other people may have a different need than yours, though.

The JSON lexer is leaps and bounds faster than the Javascript lexer and handles object keys as Name.Tag tokens instead of String.Double tokens. Adding comment highlighting doesn't affect your valid JSON files at all but allows users to highlight Microsoft's Visual Studio Code configuration files, which can have comments embedded.

[Anteru]

Right. This is a convenience feature for those people who happen to have comments in their JSON. We're not claiming that we're a validating lexer; and if you feed invalid data, anything goes. For "plain" JSON comments are invalid and thus the JSON lexer will accept all valid JSON, and then some, but not all of the JSON flavors out there. I really don't see a better way around this to be honest.