Fix #1416: add WebAssembly lexer #1564

jendrikw · 2020-10-04T10:44:14Z

This is the WebAssembly lexer I've been using and it worked quite well.

There is one test failure that I'm not sure how to resolve:

/path/pygments/.tox/lint/lib/python3.8/site-packages/pygments/lexers/webassembly.py:252: (WatLexer:arguments:pat#6) E999: Matches empty string
              (r'', Text, '#pop'),
                           ^ here

Fixes #1416.

birkenfeld · 2020-10-04T10:47:46Z

For the empty case, use default('#pop').

…regex

birkenfeld

Thanks for the PR! A few points still need addressing...

birkenfeld · 2020-10-04T10:48:37Z

pygments/lexers/webassembly.py

+            (words(builtins), Name.Builtin, 'arguments'),
+            (r'i32.const', Name.Builtin),
+            (words(['i32', 'i64', 'f32', 'f64']), Keyword.Type),
+            (r'\$[A-Za-z0-9!#$%&\'*+-./:<=>?@\\^_`|~]+', Name.Variable), # yes, all of the are valid in identifiers


+-. is suspicious, better put the dash at the end of the class.

birkenfeld · 2020-10-04T10:48:58Z

pygments/lexers/webassembly.py

+        'root': [
+            (words(keywords, suffix=r'(?=[^a-z_\.])'), Keyword),
+            (words(builtins), Name.Builtin, 'arguments'),
+            (r'i32.const', Name.Builtin),


why is this one special?

I can't remember why I did this, can be removed.

birkenfeld · 2020-10-04T10:49:30Z

pygments/lexers/webassembly.py

+i64.reinterpret_f64
+f32.reinterpret_i32
+f64.reinterpret_i64
+""".split()


please change this to a tuple display.

I'm not sure what you mean by that. Should I do them like the keywords?

yes, please, but as a tuple (with (...)). This compiles the whole list into a constant without runtime overhead.

birkenfeld · 2020-10-04T10:50:36Z

pygments/lexers/webassembly.py

+""".split()
+
+
+class WatLexer(RegexLexer):


Needs a docstring with version information (see other lexers for examples).

birkenfeld · 2020-10-04T10:51:42Z

pygments/lexers/webassembly.py

+            (r'[+-]?\d.\d(_?\d)*[eE][+-]?\d(_?\d)*', Number.Float),
+            (r'[+-]?\d.\d(_?\d)*', Number.Float),
+            (r'[+-]?\d.[eE][+-]?\d(_?\d)*', Number.Float),
+            (r'[+-]?(inf|nan|nan:0x[\dA-Fa-f](_?[\dA-Fa-f])*)', Number.Float),


The nan:xxx case should come before the nan case, otherwise it will not get matched.

birkenfeld · 2020-10-04T10:53:05Z

pygments/lexers/webassembly.py

+        'nesting_comment': [
+            (r'\(;', Comment.Multiline, '#push'),
+            (r';\)', Comment.Multiline, '#pop'),
+            (r'(.|\n)', Comment.Multiline)


this can be sped up a lot using:

(r'[^;]+', Comment.Multiline), (r';', Comment.Multiline),

I couldn't quite get this to work with nesting comments, but this does the trick:

'nesting_comment': [ (r'\(;', Comment.Multiline, '#push'), (r';\)', Comment.Multiline, '#pop'), (r'(\([^;]|;[^\)]|[^;\(])+', Comment.Multiline), ],

Ah right, I missed that comments nest. In that case it's

(r'[^;(]+', Comment.Multiline), (r'[;(]', Comment.Multiline),

Thanks, I changed it to your suggestion.

birkenfeld · 2020-10-04T10:54:22Z

pygments/lexers/webassembly.py

+        ],
+        'string': [
+            (r'\\[\dA-Fa-f][\dA-Fa-f]*', String.Escape),
+            (r'\t', String.Escape),


these are literal tab, newline and CR. did you mean r'\\t' etc?

Thanks, I tend to get confused with all the escaping.

birkenfeld · 2020-10-04T10:55:24Z

pygments/lexers/webassembly.py

+            (r"\\'", String.Escape),
+            (r'\\u\{[\dA-Fa-f](_?[\dA-Fa-f])*\}', String.Escape),
+            (r'\\\\', String.Escape),
+            (chr(92) + chr(92), Error), # backslash (double for regex)


just r'\\' would be fine, but it's better not to produce Error tokens explicitly. Just replace the final . pattern by:

(r'[^"\\]+', String.Double),

(also speeds up the matching).

birkenfeld · 2020-10-04T10:58:49Z

pygments/lexers/webassembly.py

+        ],
+        'arguments': [
+            (r'\s+', Text),
+            (r'(offset)(=)(\d(_?\d)*)', bygroups(Keyword, Operator, Number.Integer)),


This will match offset=0x..., so should get moved below the next rule. Similar with align below.

jendrikw · 2020-10-04T13:17:54Z

I think i addressed all comments.

Anteru · 2021-02-07T07:09:24Z

@birkenfeld Could you please take another look at this one?

birkenfeld · 2021-02-09T14:43:59Z

The tests need to be fixed up (output file for the example, and snippets instead of the .py file)... Otherwise looks good, just the "versionadded" is now 2.8

Anteru · 2021-02-14T08:19:11Z

Let's make this 2.9 as the target. Given this needs the tests updated, and 2.8 coming out today, no need to rush this in.

Mesteery · 2021-02-17T14:56:00Z

pygments/lexers/webassembly.py

+    and https://webassembly.github.io/spec/core/text/.
+
+
+    :copyright: Copyright 2006-2020 by the Pygments team, see AUTHORS.


Suggested change

:copyright: Copyright 2006-2020 by the Pygments team, see AUTHORS.

:copyright: Copyright 2006-2021 by the Pygments team, see AUTHORS.

Mesteery · 2021-02-17T14:56:43Z

tests/test_webassembly.py

+    Basic WatLexer Test
+    ~~~~~~~~~~~~~~~~~~~~
+
+    :copyright: Copyright 2006-2020 by the Pygments team, see AUTHORS.


Suggested change

:copyright: Copyright 2006-2020 by the Pygments team, see AUTHORS.

:copyright: Copyright 2006-2021 by the Pygments team, see AUTHORS.

Anteru · 2021-02-27T16:37:00Z

Can you please merge the latest master into this and update for the new test system? That seems to be causing the failures, and in theory all you need to do is to move your test file into a folder and add a golden test output file.

PhilippGackstatter · 2021-03-24T15:08:20Z

Hi @jendrikw, I would love to have wat highlighting and was wondering if or when you might find the time to finish this PR? Thanks a lot!

jendrikw · 2021-04-03T10:26:55Z

I thinks this is what you hand in mind. Could you please check if this can be merged?

Anteru · 2021-04-03T11:40:47Z

Pretty much yes -- nearly there :) Your unit test file should also use the new mechanism, given you only test an input for a given set of tokens. Look at tests/snippets -- you can extract the input per test into a single file, and let pytest take care of the tokens. That's easier for us because we can regenerate those without having to touch Python code.

jendrikw · 2021-04-03T12:09:45Z

That should be it.

Anteru · 2021-04-04T15:50:07Z

Merged, thanks!

add WebAssembly lexer

6754846

jendrikw changed the title ~~add WebAssembly lexer~~ Fix #1416: add WebAssembly lexer Oct 4, 2020

avoid test failure by using the default function instead of an empty …

64d2774

…regex

birkenfeld requested changes Oct 4, 2020

View reviewed changes

jendrikw added 2 commits October 4, 2020 13:48

address small issues

d05b6e7

fix WebAssembly string escapes

a017f87

change WebAssembly multiline comment parsing

0ec7f45

Anteru requested a review from birkenfeld February 7, 2021 07:09

Anteru added this to the 2.8 milestone Feb 7, 2021

Anteru removed this from the 2.8 milestone Feb 14, 2021

Anteru added this to the 2.9 milestone Feb 14, 2021

Anteru added the update needed Waiting for an update from the PR/issue creator label Feb 14, 2021

Mesteery suggested changes Feb 17, 2021

View reviewed changes

jendrikw added 2 commits February 18, 2021 09:24

update copyright year

6c0a9ec

set versionadded to 2.9

19e7e6d

jendrikw added 2 commits April 3, 2021 12:03

Merge branch 'master' into 1416-webassembly-lexer

9775cbf

change WebAssembly tests to use the new test system

140dccb

change WebAssembly unit test to use snippets

d901e2a

Anteru merged commit 06f2eba into pygments:master Apr 4, 2021

Anteru added changelog-update Items which need to get mentioned in the changelog and removed update needed Waiting for an update from the PR/issue creator changelog-update Items which need to get mentioned in the changelog labels Apr 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #1416: add WebAssembly lexer #1564

Fix #1416: add WebAssembly lexer #1564

jendrikw commented Oct 4, 2020 •

edited

Loading

birkenfeld commented Oct 4, 2020

birkenfeld left a comment

birkenfeld Oct 4, 2020

birkenfeld Oct 4, 2020

jendrikw Oct 4, 2020

birkenfeld Oct 4, 2020

jendrikw Oct 4, 2020 •

edited

Loading

birkenfeld Oct 4, 2020

birkenfeld Oct 4, 2020

birkenfeld Oct 4, 2020

birkenfeld Oct 4, 2020

jendrikw Oct 4, 2020

birkenfeld Oct 4, 2020

jendrikw Oct 6, 2020

birkenfeld Oct 4, 2020

jendrikw Oct 4, 2020

birkenfeld Oct 4, 2020

birkenfeld Oct 4, 2020

jendrikw commented Oct 4, 2020

Anteru commented Feb 7, 2021

birkenfeld commented Feb 9, 2021

Anteru commented Feb 14, 2021

Mesteery Feb 17, 2021

Mesteery Feb 17, 2021

Anteru commented Feb 27, 2021

PhilippGackstatter commented Mar 24, 2021

jendrikw commented Apr 3, 2021

Anteru commented Apr 3, 2021

jendrikw commented Apr 3, 2021

Anteru commented Apr 4, 2021

		and https://webassembly.github.io/spec/core/text/.


		:copyright: Copyright 2006-2020 by the Pygments team, see AUTHORS.

	:copyright: Copyright 2006-2020 by the Pygments team, see AUTHORS.
	:copyright: Copyright 2006-2021 by the Pygments team, see AUTHORS.

		""".split()


		class WatLexer(RegexLexer):

Fix #1416: add WebAssembly lexer #1564

Fix #1416: add WebAssembly lexer #1564

Conversation

jendrikw commented Oct 4, 2020 • edited Loading

birkenfeld commented Oct 4, 2020

birkenfeld left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jendrikw Oct 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jendrikw commented Oct 4, 2020

Anteru commented Feb 7, 2021

birkenfeld commented Feb 9, 2021

Anteru commented Feb 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anteru commented Feb 27, 2021

PhilippGackstatter commented Mar 24, 2021

jendrikw commented Apr 3, 2021

Anteru commented Apr 3, 2021

jendrikw commented Apr 3, 2021

Anteru commented Apr 4, 2021

jendrikw commented Oct 4, 2020 •

edited

Loading

jendrikw Oct 4, 2020 •

edited

Loading