Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Syntax highlighting improvements #127

gchp opened this issue Aug 5, 2016 · 2 comments

Syntax highlighting improvements #127

gchp opened this issue Aug 5, 2016 · 2 comments


Copy link

@gchp gchp commented Aug 5, 2016

The current syntax highlighting solution uses a simple lexer to generate a stream of tokens. Adding new languages means adding a new implementation of the lexer to add behaviour for that specific language. Color values are also hard coded right now.

I'd like to look into a better solution. Regular expressions is one way this could go (syntect could be interesting). Another way would be to implement some kind of AST based parser which would allow for much more granular & meaningful highlighting, I think.

Basically, I'm not sure what the best approach is, but I don't think the way it is now is it. Current solution works, but I want something better.

Comments in #88 will be useful in this.

Copy link

@abaez abaez commented Aug 5, 2016


I was researching a while ago, in making the implementation for issue #88. Just never took the time to actually do it. One thing that I did spent time in was researching and comparing implementations of grammar parsers for syntax highlighting.

Now it could be because I'm bias to textadept's way, but after looking over how vim and sublime text have gone about it, I've come to the realization that using an AST parser defined by a PEG generator would be best. Right now there's two generators written in rust, which come to mind for the process; rusty-peg and oak. These two would allow to get away with making a lexer definition which is highly expressive, with leaner symantics, and less of a resource hog for the editor at hand.

The reason why I bring this up is for the ease in definitions of a lexer for Iota and the ease in implementing the parser should both be simple enough to refactor and modify by contributors. Using a PEG generator would allow very easy definitions of lexers. And I've noticed that PEG generated lexers, like scintillua, can be less daunting for the editor at hand.

In any case, here's an old talk about the power of PEG generator, which is still relevant to the issue at hand and can probably speak better on the advantage over regex parser.

Copy link

@luciusmagn luciusmagn commented Apr 30, 2017

Have you seen the way syntax highlighting works in Nano? Extremely simple but quite effective. With a few improvements, it could work nicely. It is regex based

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants