Please add a JSON5 lexer

[purple512]

JSON5 specification - a superset of JSON

It would be nice to be able to format JSON5 files.

[ct456]

Hi, I'm a student from the University of Adelaide, I'm in a group of students working on contributing to this github repository. We'd like to take on this issue if its still available.

[jeanas]

Feel free. Since JSON5 looks like a not-too-big extension to JSON, I'd recommend extending the existing JSON lexer. Out of curiosity, who else is there in your student group? Is your university asking you to do this?

[ct456]

Hi, thank you for the prompt reply and tips. I'll be the one primarily working on the code, there are other members who will be working on other non-git related work. And yes this is part of an assessment piece for a subject.

[BuildMonkey]

Hello @jean-abou-samra,

I am the lecturer for the course. This is a student team working on this issue. The team has been notified that they will need to collaborate on code contributions.

We are from the University of Adelaide, where students have an eight-week project contributing to an OSS project. Having briefly reading this issue, it seems like this might not be a good issue for a team to work on over weeks.

Perhaps there is another issue that might be better suited for a team in the time period provided.

Regards,
Dr. Rita Garcia

[jeanas]

The bulk of Pygments is lexers for various languages. I think working on lexers is not a bad idea, as it will make for nicely self-contained projects. On the other hand, if it's a team of five and a project of eight weeks, just one such issue might not be enough. Luckily, there are dozens of them in the tracker ...

I think the project this team could tackle really depends on who they are. @ct456 What is your academic level? Did any of you contribute to open source software before? Is it your first year with programming or are you more advanced? Do you study computer science as a main subject or as part of a curriculum in another field? Do you have some experience of version control (whether Git, Bazaar, SVN, ...)?

Besides improving all the lexers, here are a few items I can think of, which are not necessarily captured in tracker issues (EDIT: maybe @Anteru and @birkenfeld have more to suggest):

• Improving tests. Under scripts/, you can find a few tests that have not yet been made required, and would ideally be, fixing existing failures as much as possible or (if not feasible all at once) adding lists of known failures. This can seem less attractive because it doesn't directly impact users, but for development tests are paramount.
• Adding new styles or improving existing formatters (the HTML one for example has some feature requests in the tracker).
• Designing a more general mechanism for handling lists of builtins, perhaps with a JSON structure loaded from a file and passed as a parameter to the lexer, so that it will be easier to keep lists of builtins in sync with the evolution of each of the programming languages.
• Improving regexlint, the tool we use to catch issues in regular expressions. My personal dream would be a check for catastrophic backtracking, which is a common issue affecting regexes (see the file CONTRIBUTING.md in the source) that can be easy to miss or give headaches to track. It would be very helpful to have a tool catching them automatically. I'd consider this a feasible project for CS students with a good background in algorithms and formal languages (but I have not experimented with it to see how hard it is).
• For a group of master's students working full weeks: reimplement language recognition. Currently, lexers have an analyse_text method that returns a float between 0 and 1 indicating if the text is likely to be a program in the language in question. This isn't very reliable, see Refactor/Rewrite analyze_text #1702. There could be a mechanism based on AI trained with language data, on reusing the lexers' regexes, or a mix of both, to improve reliability. Of course this is a very ambitious target.

You will soon understand that while you will be working on this project for your university (at least at first, I hope you'll keep contributing afterwards :-), virtually everyone else is working on it for pleasure and fun. I recommend picking an issue that motivates you (of course, as long as it is also a reasonably sized goal for your allotted time and your current programming level). Also, feel free to ask any questions. I am myself a third-year math and CS student, so I've been in your shoes (first OSS contribution) just two years ago.

[BuildMonkey]

Thanks @jean-abou-samra for your help on this matter.

I believe we found another issue that the team can achieve and is comparable to another team working on lexer.

[ct456]

Sorry about the trouble @jean-abou-samra, and thank you for being so helpful and patient with us. We've found another issue regarding pygments that we can work on as mentioned by Dr. Garcia above, thank you again.