Lexer request: add jsonc lexer

[Mesteery]

Add a JSONC lexer, a basic JSON with comments (// inline and /* block */).

[Anteru]

@kurtmckee Does this sound like something you could look at?

[kurtmckee]

Yep! I saw the request earlier this week and mulled over whether to make this part of the default JSON lexer or not. Haven't reached a conclusion yet, though; I need to review the JSONC spec a little more.

[kurtmckee]

After review, it appears that every JSONC proposal (and there are multiple) is just an attempt to make a subset of Javascript into a superset of JSON.

In other words, Pygments' Javascript lexer appears to be more than sufficient to parse JSON with comments added.

1. The Javascript lexer already parses JSON with comments.
2. No single proposal going by the name "JSONC" is a clear winner of the term.

For this reason, I recommend closing this ticket and directing users to use the full Javascript lexer.

[Anteru]

Perfect, thanks for checking. I wasn't aware you handle comments already -- and yes, it totally makes sense to not cater for various JSONC flavors if all they add are comments.

[alainpannetier]

Does that work?

{
"a" : "b"
}
Is single colour in javascript and two-colours ("key": "value") in json.

So javascript cannot work for json.

[kurtmckee]

@alainpannetier, you're correct that the object keys and string values are a single color in Javascript. This is because the Javascript lexer interprets anything in double quotes as a String.Double token. However, that does not mean that the Javascript lexer cannot be used for JSON. It just means the Javascript lexer could be improved so that it treats object keys as Name.Tag tokens.

As I wrote above, every specification that introduces comments in JSON (and there are many!) just introduces features that are already handled by the Javascript parser. Whether it's called "JSONC", "JSON-C", or "JSON5", it appears that each specification introduces comments and other things that make them something more than JSON...but something less than Javascript, which already supports all of their proposed features.

┌──────────────────────────────────┐
│            Javascript            │
│                                  │
│  ┌────────────────────────────┐  │
│  │    JSONC, JSON-C, JSON5    │  │
│  │                            │  │
│  │          ┌──────┐          │  │
│  │          │ JSON │          │  │
│  │          └──────┘          │  │
│  │                            │  │
│  └────────────────────────────┘  │
│                                  │
└──────────────────────────────────┘

My recommendation is to target an improvement to the Javascript parser.

[kurtmckee]

I've added comment support to JSON in #2049. It's not part of the JSON spec but adding it doesn't affect parsing of valid JSON documents.