Add Solidity (lexer and examples) #760

veox · 2017-08-22T14:47:04Z

Adds a lexer for the Solidity language. It's based on Rouge's C lexer.

The longer example in sample is from my Pygments lexer package. The shorter one in demos was written specifically for Rouge.

Compared to the linked Pygments lexer, the one here has ~~more~~ less up-to-date keywords, ~~but~~ and doesn't have a distinct assembly state (yet). This is not a big issue: the low-level assembly keywords (e.g. let) are listed (in self.reserved).

Here's the demo file on Lightshow, for comparison with a yet another highlighter (using davidhq/SublimeEthereum grammar file).

I've squashed the development history into one commit; do tell if this is undesirable.

Solidity is a high(?)-level language for the Ethereum Virtual Machine (EVM), a blockchain-based "decentralised computer".

More info: solc compiler, its docs; ethereum.org landing page

This fixes #987.

veox · 2017-08-22T15:03:29Z

To be transparent: Solidity's not in github's linguist (yet), and looks like it won't be for the foreseeable future.

My hopes with adding it here is that GitLab will eventually pick it up with a Rouge update; and the Solidity community will finally get a little-effort way of having highlighted code online.

spec/lexers/solidity_spec.rb

lib/rouge/lexers/solidity.rb

Arachnid · 2017-11-13T14:00:09Z

Any chance of this getting merged? It'd be really handy.

lib/rouge/lexers/solidity.rb

veganstraightedge · 2018-02-01T06:56:48Z

What's the over/under that this'll get merged in, @jneen? I'd love to be able to use Rouge and stay in Ruby Land instead of farming out to Pygments.

PS. Thanks for making Rouge! ❤️

pklinger · 2019-03-21T20:01:36Z

Is there any chance this will ever get merged or what is holding this back @jneen @dblessing ?

FabioBonfiglio · 2019-11-08T15:55:26Z

Would love to see this merged too...
🙏

axic · 2019-12-08T22:44:26Z

To be transparent: Solidity's not in github's linguist (yet), and looks like it won't be for the foreseeable future.

Actually linguist merged the PR on 12th Jan 2018: github-linguist/linguist#3973

axic · 2020-03-23T12:27:40Z

@jneen @dblessing @pyrmont @gfx can you give some advice what would be needed to get his PR considered for merging?

It definitely needs to be updated for latest syntax.

pyrmont · 2020-03-24T17:42:55Z

@axic I'm sorry this has been outstanding for such a long time. I have a couple of comments to make but is @veox still monitoring this? (I wouldn't blame you if you weren't, @veox.) Just wondering whether it makes more sense for someone else to submit a new PR that they can be responsible for or not.

chriseth · 2020-03-25T17:31:43Z

@pyrmont thanks for your quick response! I'll add some comments where this needs to be updated to the latest version of the language if you don't mind. If veox is not working on this anymore, we can open a new PR any time.

lib/rouge/lexers/solidity.rb

veox · 2020-03-25T19:39:39Z

I'll get to this tomorrow, a bit too late today.

(The Pygments highlighter has been updated to Solidity v0.6.0 syntax, so this should be relatively easy.)

axic · 2020-03-25T19:51:59Z

Thanks @pyrmont for the answer! @veox let me know if you need any help.

Squashed commits: spec/lexers: Solidity spec, based on tutorial's Turtle spec. samples: add Solidity sample (from pygments-lexer-solidity). See https://gitlab.com/veox/pygments-lexer-solidity. File permalink: https://gitlab.com/veox/pygments-lexer-solidity/blob/8e9919b5e9e91a2022def2a4c51003bc99042686/example.sol The Pygments lexer's example was written by me, and is licensed under a BSD license. demos: minimal Solidity demo. lexers: track solidity.rb (verbatim copy of c.rb). lexers/solidity: proper class, extensions, other meta stuff. lexers/solidity: added some keywords (not all!). lexers/solidity: remove float, added types TODO. lexers/solidity: constants, strings. lexers/solidity: clean-up + remove function state. lexers/solidity: complex MxN types. lexers: fix mimetype (`text/solidity` -> `text/x-solidity`). spec: comment guess-by-source section that makes tests fail. Test fails because something (?) is guessed as PlainText. lexers/solidity: add missing `assert` and `require` keywords (+ demos/solidity: rewrite). A few more keywords may well be missing. Used https://github.com/ethereum/solidity/blob/0a04a35a2e02437ae038af41c947b3e829946bca/libsolidity/parsing/Token.h to check - probably should look for "preprocessor" directives, or built-ins, or the like, - in Solidity's source tree. demos/solidity: fix to compile. lexers/solidity: populate builtins + remove a few more C lexer leftovers. SQ

…first.

This is done before "porting" in changes from Pygments' lexer, so it's out of the way.

These are "ported" from Pygments' lexer. I'm not 100% on the terminology.

Natspec, Assembly, Yul, and some built-in functions are still unimplemented.

This calms down Rake warnings about ambivalence.

pyrmont

Two comments:

lib/rouge/lexers/solidity.rb

…, N.

They are already covered in `def self.builtins`.

pyrmont

Just one final question, I think:

lib/rouge/lexers/solidity.rb

pyrmont

Sorry this is dragging out a bit :(

lib/rouge/lexers/solidity.rb

pyrmont

@veox Sorry about the comment before. I realise now what you were going to for with the open comment rule. I've tried a different solution. What do you think?

lib/rouge/lexers/solidity.rb

lib/rouge/demos/solidity

lib/rouge/lexers/solidity.rb

Outline also why there should be no multi-level comments, demonstrate it, and allow for reasonably bloodless manual testing by removing spaces.

@axic

Thanks @axic! Co-Authored-By: Alex Beregszaszi <alex@rtfs.hu>

veox · 2020-04-05T15:03:36Z

Argh! The check was cancelled. >_<

pyrmont · 2020-04-05T15:34:24Z

@veox No problem. I've rerun the CI test and it's all green now. Will merge this into master. Thanks for the submission!

spec/visual/samples/solidity

Two visual demo cases got conflated: the initial one of multiline comments being open until the end of file, and the new one of no such thing as nested multiline comment blocks. This commits separates them.

pyrmont · 2020-04-05T20:01:33Z

Thanks for the effort, everyone. This will be part of the next release of Rouge. We have that scheduled to go out on Tuesday 14 April 🎉

axic · 2020-04-05T20:04:51Z

Thanks @veox and @pyrmont for getting this through!

This commit adds a lexer for Solidity.

veox changed the title ~~spec/lexers/sampes/demos: add Solidity lexer (squashed branch).~~ Add Solidity (lexer and examples) Aug 22, 2017

veox commented Aug 22, 2017

View reviewed changes

spec/lexers/solidity_spec.rb Outdated Show resolved Hide resolved

jneen reviewed Aug 22, 2017

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

holiman reviewed Nov 13, 2017

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

veox mentioned this pull request Mar 4, 2018

Add new keywords to syntax highlighters ethereum/solidity#3637

Closed

2 tasks

ritzdorf mentioned this pull request Sep 11, 2018

Add support for Solidity #987

Closed

axic mentioned this pull request Dec 8, 2019

Add solidity highlighting support to rouge ethereum/solidity#7920

Closed

pyrmont self-assigned this Mar 24, 2020

pyrmont added the needs-review The PR needs to be reviewed label Mar 24, 2020

chriseth reviewed Mar 25, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

chriseth reviewed Mar 25, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

veox added 7 commits March 26, 2020 19:13

lexers/solidity: don't map/merge sets on every invocation - just the …

672ee0a

…first.

lexers/solidity: add new keywords mentioned in feedback.

d5f6378

This is done before "porting" in changes from Pygments' lexer, so it's out of the way.

lexers/solidity: add changes up to (mostly) Solidity v0.4.22.

e1b6522

These are "ported" from Pygments' lexer. I'm not 100% on the terminology.

lexers/solidity: add new keywords from Solidity v0.6.0.

e65c995

Natspec, Assembly, Yul, and some built-in functions are still unimplemented.

samples/solidity: copy updated sample over from Pygments.

612c65b

lexers/solidity: mark regexes in rules explicitly.

dbec54a

This calms down Rake warnings about ambivalence.

veox force-pushed the add-lexer-solidity-squashed branch from 37dd378 to dbec54a Compare March 26, 2020 17:23

pyrmont suggested changes Apr 2, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

veox added 3 commits April 3, 2020 15:38

lexers/solidity: [u]fixed{M}x{N} should have at least one digit for M…

6f793ba

…, N.

lexers/solidity: use regex for bytes{N} and [u]int{N}, too.

2242a4d

lexers/solidity: remove duplicate rule for (block|msg|tx).{stuff}.

d7ccc70

They are already covered in `def self.builtins`.

veox requested a review from pyrmont April 3, 2020 12:48

pyrmont reviewed Apr 3, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

veox added 2 commits April 4, 2020 17:56

lexers/solidity: catch case of unclosed multi-line comment.

f19341e

spec/solidity: update with less incorrect language syntax.

8b42435

pyrmont reviewed Apr 5, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

pyrmont added 3 commits April 5, 2020 15:04

Add nested comments

6d46215

Add state for multiline comments

c8d3f3f

Simplify demo

2e824fa

pyrmont reviewed Apr 5, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

veox commented Apr 5, 2020

View reviewed changes

lib/rouge/demos/solidity Outdated Show resolved Hide resolved

pyrmont added 2 commits April 5, 2020 20:49

Remove nested closing comments

b56f33c

Remove nesting rule from comment state

479a6d2

axic reviewed Apr 5, 2020

View reviewed changes

lib/rouge/lexers/solidity.rb Outdated Show resolved Hide resolved

veox and others added 2 commits April 5, 2020 17:39

samples/solidity: make example of comment-to-EOF more wordy.

8b471bb

Outline also why there should be no multi-level comments, demonstrate it, and allow for reasonably bloodless manual testing by removing spaces.

lexers/solidity: add missing abi.decode.

103718e

Thanks @axic! Co-Authored-By: Alex Beregszaszi <alex@rtfs.hu>

pyrmont reviewed Apr 5, 2020

View reviewed changes

spec/visual/samples/solidity Outdated Show resolved Hide resolved

samples/solidity: split no-multiline-nesting and multiline-to-eof cases.

1555984

Two visual demo cases got conflated: the initial one of multiline comments being open until the end of file, and the new one of no such thing as nested multiline comment blocks. This commits separates them.

pyrmont approved these changes Apr 5, 2020

View reviewed changes

pyrmont merged commit 561a429 into rouge-ruby:master Apr 5, 2020

pyrmont removed the author-action The PR has been reviewed but action by the author is needed label Apr 5, 2020

chriseth mentioned this pull request Apr 16, 2020

Make syntax highlighting work for the blog. ethereum/solidity-blog#34

Closed

mattt pushed a commit to NSHipster/rouge that referenced this pull request May 21, 2020

Add Solidity lexer (rouge-ruby#760)

9fcee31

This commit adds a lexer for Solidity.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Solidity (lexer and examples) #760

Add Solidity (lexer and examples) #760

veox commented Aug 22, 2017 •

edited by pyrmont

veox commented Aug 22, 2017

Arachnid commented Nov 13, 2017

veganstraightedge commented Feb 1, 2018

pklinger commented Mar 21, 2019 •

edited

FabioBonfiglio commented Nov 8, 2019

axic commented Dec 8, 2019

axic commented Mar 23, 2020

pyrmont commented Mar 24, 2020

chriseth commented Mar 25, 2020

veox commented Mar 25, 2020

axic commented Mar 25, 2020

pyrmont left a comment

pyrmont left a comment

pyrmont left a comment •

edited

pyrmont left a comment •

edited

veox commented Apr 5, 2020

pyrmont commented Apr 5, 2020

pyrmont commented Apr 5, 2020

axic commented Apr 5, 2020

Add Solidity (lexer and examples) #760

Add Solidity (lexer and examples) #760

Conversation

veox commented Aug 22, 2017 • edited by pyrmont

veox commented Aug 22, 2017

Arachnid commented Nov 13, 2017

veganstraightedge commented Feb 1, 2018

pklinger commented Mar 21, 2019 • edited

FabioBonfiglio commented Nov 8, 2019

axic commented Dec 8, 2019

axic commented Mar 23, 2020

pyrmont commented Mar 24, 2020

chriseth commented Mar 25, 2020

veox commented Mar 25, 2020

axic commented Mar 25, 2020

pyrmont left a comment

Choose a reason for hiding this comment

pyrmont left a comment

Choose a reason for hiding this comment

pyrmont left a comment • edited

Choose a reason for hiding this comment

pyrmont left a comment • edited

Choose a reason for hiding this comment

veox commented Apr 5, 2020

pyrmont commented Apr 5, 2020

pyrmont commented Apr 5, 2020

axic commented Apr 5, 2020

veox commented Aug 22, 2017 •

edited by pyrmont

pklinger commented Mar 21, 2019 •

edited

pyrmont left a comment •

edited

pyrmont left a comment •

edited