Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enh(elixir) Improves and fixes many issues with Elixir grammar #3212

Merged
merged 11 commits into from
Jun 5, 2021
3 changes: 3 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@ Grammars:
- fix(python) identifiers starting with underscore not highlighted (#3221) [Antoine Lambert][]
- enh(clojure) added `edn` alias (#3213) [Stel Abrego][]
- enh(elixir) much improved regular expression sigil support (#3207) [Josh Goebel][]
- enh(elixir) updated list of keywords (#3212) [Angelika Tyborska][]
- fix(elixir) fixed number detection when numbers start with a zero (#3212) [Angelika Tyborska][]
Comment on lines +8 to +9
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming the id should be the PR id, not the issue id?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep


[Stel Abrego]: https://github.com/stelcodes
[Josh Goebel]: https://github.com/joshgoebel
[Antoine Lambert]: https://github.com/anlambert
[Angelika Tyborska]: https://github.com/angelikatyborska

## Version 11.0.0

Expand Down
44 changes: 22 additions & 22 deletions src/languages/elixir.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,46 +13,45 @@ export default function(hljs) {
const ELIXIR_IDENT_RE = '[a-zA-Z_][a-zA-Z0-9_.]*(!|\\?)?';
const ELIXIR_METHOD_RE = '[a-zA-Z_]\\w*[!?=]?|[-+~]@|<<|>>|=~|===?|<=>|[<>]=?|\\*\\*|[-/+%^&*~`|]|\\[\\]=?';
const KEYWORDS = [
"alias",
"after",
"alias",
"and",
"begin",
"break",
"case",
"catch",
"cond",
"defined",
"defstruct",
"do",
"else",
"end",
"ensure",
"false",
"fn",
"for",
"if",
"import",
"in",
"include",
"module",
"next",
"nil",
"not",
"or",
"quote",
"redo",
"raise",
"receive",
"require",
"retry",
"return",
"self",
"then",
"true",
"reraise",
"rescue",
"try",
"unless",
"until",
"unquote",
"unquote_splicing",
"use",
"when",
"while",
"with|0"
];
const LITERALS = [
"false",
"nil",
"true"
];
const KWS = {
$pattern: ELIXIR_IDENT_RE,
keyword: KEYWORDS
keyword: KEYWORDS,
literal: LITERALS
};
const SUBST = {
className: 'subst',
Expand All @@ -62,7 +61,7 @@ export default function(hljs) {
};
const NUMBER = {
className: 'number',
begin: '(\\b0o[0-7_]+)|(\\b0b[01_]+)|(\\b0x[0-9a-fA-F_]+)|(-?\\b[1-9][0-9_]*(\\.[0-9_]+([eE][-+]?[0-9]+)?)?)',
begin: '(\\b0o[0-7_]+)|(\\b0b[01_]+)|(\\b0x[0-9a-fA-F_]+)|(-?\\b[0-9][0-9_]*(\\.[0-9_]+([eE][-+]?[0-9]+)?)?)',
relevance: 0
};
// TODO: could be tightened
Expand Down Expand Up @@ -218,7 +217,7 @@ export default function(hljs) {
};
const FUNCTION = {
className: 'function',
beginKeywords: 'def defp defmacro',
beginKeywords: 'def defp defmacro defmacrop',
end: /\B\b/, // the mode is ended by the title
contains: [
hljs.inherit(hljs.TITLE_MODE, {
Expand Down Expand Up @@ -272,6 +271,7 @@ export default function(hljs) {

return {
name: 'Elixir',
aliases: ['ex', 'exs'],
joshgoebel marked this conversation as resolved.
Show resolved Hide resolved
keywords: KWS,
contains: ELIXIR_DEFAULT_CONTAINS
};
Expand Down
40 changes: 40 additions & 0 deletions test/markup/elixir/conditionals.expect.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<span class="hljs-keyword">case</span> x <span class="hljs-keyword">do</span>
<span class="hljs-number">1</span> -&gt; <span class="hljs-symbol">:one</span>
<span class="hljs-number">2</span> -&gt; <span class="hljs-symbol">:two</span>
_ -&gt; <span class="hljs-symbol">:error</span>
<span class="hljs-keyword">end</span>

<span class="hljs-keyword">cond</span> <span class="hljs-keyword">do</span>
x &gt; <span class="hljs-number">30</span> -&gt; <span class="hljs-symbol">:ok</span>
y &lt;= <span class="hljs-number">7</span> -&gt; <span class="hljs-symbol">:maybe</span>
z == <span class="hljs-symbol">:skip</span> -&gt; <span class="hljs-symbol">:ok</span>
<span class="hljs-literal">true</span> -&gt; <span class="hljs-symbol">:error</span>
<span class="hljs-keyword">end</span>

<span class="hljs-keyword">if</span> x &gt; <span class="hljs-number">4</span> <span class="hljs-keyword">do</span>
<span class="hljs-symbol">:ok</span>
<span class="hljs-keyword">end</span>

<span class="hljs-keyword">if</span> x &gt; <span class="hljs-number">4</span>, <span class="hljs-symbol">do:</span> <span class="hljs-symbol">:ok</span>

<span class="hljs-keyword">if</span> x &gt; <span class="hljs-number">4</span> <span class="hljs-keyword">do</span>
<span class="hljs-symbol">:ok</span>
<span class="hljs-keyword">else</span>
<span class="hljs-symbol">:error</span>
<span class="hljs-keyword">end</span>

<span class="hljs-keyword">if</span> x &gt; <span class="hljs-number">4</span>, <span class="hljs-symbol">do:</span> <span class="hljs-symbol">:ok</span>, <span class="hljs-symbol">else:</span> <span class="hljs-symbol">:error</span>

<span class="hljs-keyword">unless</span> y &lt; <span class="hljs-number">50</span> <span class="hljs-keyword">do</span>
<span class="hljs-symbol">:error</span>
<span class="hljs-keyword">end</span>

<span class="hljs-keyword">unless</span> y &lt; <span class="hljs-number">50</span>, <span class="hljs-symbol">do:</span> <span class="hljs-symbol">:error</span>

<span class="hljs-keyword">unless</span> y &lt; <span class="hljs-number">50</span> <span class="hljs-keyword">do</span>
<span class="hljs-symbol">:error</span>
<span class="hljs-keyword">else</span>
<span class="hljs-symbol">:ok</span>
<span class="hljs-keyword">end</span>

<span class="hljs-keyword">unless</span> y &lt; <span class="hljs-number">50</span>, <span class="hljs-symbol">do:</span> <span class="hljs-symbol">:error</span>, <span class="hljs-symbol">else:</span> <span class="hljs-symbol">:ok</span>
40 changes: 40 additions & 0 deletions test/markup/elixir/conditionals.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
case x do
1 -> :one
2 -> :two
_ -> :error
end

cond do
x > 30 -> :ok
y <= 7 -> :maybe
z == :skip -> :ok
true -> :error
end

if x > 4 do
:ok
end

if x > 4, do: :ok

if x > 4 do
:ok
else
:error
end

if x > 4, do: :ok, else: :error

unless y < 50 do
:error
end

unless y < 50, do: :error

unless y < 50 do
:error
else
:ok
end

unless y < 50, do: :error, else: :ok
6 changes: 6 additions & 0 deletions test/markup/elixir/function-title.expect.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,10 @@

<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">f!</span></span>, <span class="hljs-symbol">do:</span> IO.puts <span class="hljs-string">&quot;hello world&quot;</span>

<span class="hljs-function"><span class="hljs-keyword">defp</span> <span class="hljs-title">f?</span></span>, <span class="hljs-symbol">do:</span> <span class="hljs-literal">true</span>

<span class="hljs-function"><span class="hljs-keyword">defmacro</span> <span class="hljs-title">foo</span></span>, <span class="hljs-symbol">do:</span> <span class="hljs-symbol">:ok</span>

<span class="hljs-function"><span class="hljs-keyword">defmacrop</span> <span class="hljs-title">do_foo</span></span>, <span class="hljs-symbol">do:</span> <span class="hljs-symbol">:ok</span>

x = <span class="hljs-number">5</span>
6 changes: 6 additions & 0 deletions test/markup/elixir/function-title.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,10 @@ end

def f!, do: IO.puts "hello world"

defp f?, do: true

defmacro foo, do: :ok

defmacrop do_foo, do: :ok

x = 5
12 changes: 12 additions & 0 deletions test/markup/elixir/modules.expect.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<span class="hljs-class"><span class="hljs-keyword">defmodule</span> <span class="hljs-title">User</span></span> <span class="hljs-keyword">do</span>
<span class="hljs-keyword">defstruct</span> [<span class="hljs-symbol">:name</span>, <span class="hljs-symbol">:email</span>, <span class="hljs-symbol">age:</span> <span class="hljs-number">18</span>]
<span class="hljs-keyword">end</span>

<span class="hljs-class"><span class="hljs-keyword">defprotocol</span> <span class="hljs-title">Size</span></span> <span class="hljs-keyword">do</span>
<span class="hljs-variable">@doc</span> <span class="hljs-string">&quot;Calculates the size (and not the length!) of a data structure&quot;</span>
<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">size</span></span>(data)
<span class="hljs-keyword">end</span>

<span class="hljs-class"><span class="hljs-keyword">defimpl</span> <span class="hljs-title">Size</span></span>, <span class="hljs-symbol">for:</span> Map <span class="hljs-keyword">do</span>
<span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">size</span></span>(map), <span class="hljs-symbol">do:</span> map_size(map)
<span class="hljs-keyword">end</span>
12 changes: 12 additions & 0 deletions test/markup/elixir/modules.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
defmodule User do
defstruct [:name, :email, age: 18]
end

defprotocol Size do
@doc "Calculates the size (and not the length!) of a data structure"
def size(data)
end

defimpl Size, for: Map do
def size(map), do: map_size(map)
end
7 changes: 7 additions & 0 deletions test/markup/elixir/numbers.expect.txt
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
integer = <span class="hljs-number">1234</span>
integer_with_leading_zero = <span class="hljs-number">01234</span>
integer_zero = <span class="hljs-number">0</span>
big_integer = <span class="hljs-number">1_234_000</span>
neg_integer = <span class="hljs-number">-20_000</span>
float = <span class="hljs-number">2.34</span>
float_with_leading_zero = <span class="hljs-number">0.34</span>
float_zero = <span class="hljs-number">0.0</span>
sci_float = <span class="hljs-number">2.4e23</span>
plus_sci_float = <span class="hljs-number">2.4e+23</span>
small_sci_float = <span class="hljs-number">2.4e-23</span>
cap_sci_float = <span class="hljs-number">2.4E23</span>
binary = <span class="hljs-number">0b1010</span>
binary_with_leading_zero = <span class="hljs-number">0b0010</span>
strange_binary = <span class="hljs-number">0b1010_1010_1010</span>
octal = <span class="hljs-number">0o777</span>
octal_with_leading_zero = <span class="hljs-number">0o077</span>
strange_octal = <span class="hljs-number">0o777_666_555</span>
hex = <span class="hljs-number">0x1ABEF</span>
hex_with_leading_zero = <span class="hljs-number">0x0ABEF</span>
strange_hex = <span class="hljs-number">0x1234_FACE_987D</span>
7 changes: 7 additions & 0 deletions test/markup/elixir/numbers.txt
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
integer = 1234
integer_with_leading_zero = 01234
integer_zero = 0
big_integer = 1_234_000
neg_integer = -20_000
float = 2.34
float_with_leading_zero = 0.34
float_zero = 0.0
sci_float = 2.4e23
plus_sci_float = 2.4e+23
small_sci_float = 2.4e-23
cap_sci_float = 2.4E23
binary = 0b1010
binary_with_leading_zero = 0b0010
strange_binary = 0b1010_1010_1010
octal = 0o777
octal_with_leading_zero = 0o077
strange_octal = 0o777_666_555
hex = 0x1ABEF
hex_with_leading_zero = 0x0ABEF
strange_hex = 0x1234_FACE_987D
8 changes: 6 additions & 2 deletions test/markup/elixir/strings.expect.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
a = <span class="hljs-string">&quot;&quot;&quot;test&quot;&quot;&quot;</span>
b = <span class="hljs-string">&#x27;&#x27;&#x27;test&#x27;&#x27;&#x27;</span>
a = <span class="hljs-string">&quot;&quot;&quot;
test
&quot;&quot;&quot;</span>
b = <span class="hljs-string">&#x27;&#x27;&#x27;
test
&#x27;&#x27;&#x27;</span>
c = <span class="hljs-string">&quot;test&quot;</span>
d = <span class="hljs-string">&#x27;test&#x27;</span>
8 changes: 6 additions & 2 deletions test/markup/elixir/strings.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
a = """test"""
b = '''test'''
Copy link
Member

@joshgoebel joshgoebel May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see how these are better examples (that you added), but if the previous were valid we should also leave them to serve as additional regression tests... one can still use a multi-line string for a single line - that is valid grammar, yes?

Copy link
Contributor Author

@angelikatyborska angelikatyborska May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is valid grammar, yes?

No, those were syntax errors in Elixir. The only allowed characters after the opening '''/""" is optional whitespace, and then a newline is required.

My assumption is that it's not a big deal if the syntax highlighting colors something that doesn't compile so I didn't feel the need to change the regexes for multiline strings, but the test examples should be valid Elixir code, right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, those were syntax errors in Elixir.

Whoa. I stand corrected. ;-)

My assumption is that it's not a big deal if the syntax highlighting colors something that doesn't compile

Right, it's not our job to detect invalid code.

but the test examples should be valid Elixir code, right?

For the most part, yes. In this particular case this indeed makes sense. I just see people "swap" test cases often when they should be adding new cases instead, [not removing old ones]. Thought this was perhaps yet another case of that. :)

a = """
test
"""
b = '''
test
'''
c = "test"
d = 'test'