Double newlines in HTML formatter after Elixir comments

[dbernheisel]

When formatting newline whitespace in HTML inside a <pre>, newlines are sometimes rendered twice.

• First the original newline token is rendered, and then
• the token has another newline to close the span.

This only seems to happen after comments in Elixir code, so this issue might be a tokenizer issue and not an HTML formatter issue.

---

For example, using makeup 1.0.5 and makeup_elixir 0.15.1, given this string:

defmodule BestStructEver do\r\n  defstruct [:a]\r\nend\r\n\r\n# Struct to simple map:\r\niex> Map.from_struct(%BestStructEver{a: \"foobar\"})\r\n# => %{a: \"foobar\"}\r\n\r\n# Map to a struct:\r\niex> struct(BestStructEver, %{a: \"foobar\"})\r\n# => %BestStructEver{a: \"foobar\"}\r\n\r\n# Note: The struct function is from Kernel, so `Kernel.` can be omitted.

It is rendered this way using a Phoenix template <%= raw(Makeup.highlight(my_code_here)) %>:

<pre class="highlight"><code><span class="kd">defmodule</span><span class="w"> </span><span class="nc">BestStructEver</span><span class="w"> </span><span class="k" data-group-id="8518303302-1">do</span><span class="w">
  </span><span class="kd">defstruct</span><span class="w"> </span><span class="p" data-group-id="8518303302-2">[</span><span class="ss">:a</span><span class="p" data-group-id="8518303302-2">]</span><span class="w">
</span><span class="k" data-group-id="8518303302-1">end</span><span class="w">

</span><span class="c1"># Struct to simple map:
</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="nc">Map</span><span class="o">.</span><span class="n">from_struct</span><span class="p" data-group-id="8518303302-3">(</span><span class="p" data-group-id="8518303302-4">%</span><span class="nc" data-group-id="8518303302-4">BestStructEver</span><span class="p" data-group-id="8518303302-4">{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="s">"foobar"</span><span class="p" data-group-id="8518303302-4">}</span><span class="p" data-group-id="8518303302-3">)</span><span class="w">
</span><span class="c1"># =&gt; %{a: "foobar"}
</span><span class="w">

</span><span class="c1"># Map to a struct:
</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">struct</span><span class="p" data-group-id="8518303302-5">(</span><span class="nc">BestStructEver</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8518303302-6">%{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="s">"foobar"</span><span class="p" data-group-id="8518303302-6">}</span><span class="p" data-group-id="8518303302-5">)</span><span class="w">
</span><span class="c1"># =&gt; %BestStructEver{a: "foobar"}
</span><span class="w">

</span><span class="c1"># Note: The struct function is from Kernel, so `Kernel.` can be omitted.</span></code></pre>

appearing like this image:

When it should appear more like this (ignoring the theme -- this is not Makeup):

---

When I fork makeup, I think I fix it if I treat the newline as if it needed escaping, rendering an empty string instead, but I feel like this fixes it the wrong way. Instead I feel the span closing tag should not have a newline before it.

diff --git a/lib/makeup/formatters/html/html_formatter.ex b/lib/makeup/formatters/html/html_formatter.ex
index 1c28d51..912f30e 100644
--- a/lib/makeup/formatters/html/html_formatter.ex
+++ b/lib/makeup/formatters/html/html_formatter.ex
@@ -38,6 +38,8 @@ defmodule Makeup.Formatters.HTML.HTMLFormatter do
     render_token(escaped_value, css_class, meta, highlight_tag)
   end
 
+  defp escape_for(?\n), do: ""
+
   defp escape_for(?&), do: "&"
 
   defp escape_for(?<), do: "&lt;"

It produces this HTML:

<pre class="highlight"><code><span class="kd">defmodule</span><span class="w"> </span><span class="nc">BestStructEver</span><span class="w"> </span><span class="k" data-group-id="7157529561-1">do</span><span class="w">
  </span><span class="kd">defstruct</span><span class="w"> </span><span class="p" data-group-id="7157529561-2">[</span><span class="ss">:a</span><span class="p" data-group-id="7157529561-2">]</span><span class="w">
</span><span class="k" data-group-id="7157529561-1">end</span><span class="w">

</span><span class="c1"># Struct to simple map:
</span><span class="w"></span><span class="gp unselectable">iex&gt; </span><span class="nc">Map</span><span class="o">.</span><span class="n">from_struct</span><span class="p" data-group-id="7157529561-3">(</span><span class="p" data-group-id="7157529561-4">%</span><span class="nc" data-group-id="7157529561-4">BestStructEver</span><span class="p" data-group-id="7157529561-4">{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="s">"foobar"</span><span class="p" data-group-id="7157529561-4">}</span><span class="p" data-group-id="7157529561-3">)</span><span class="w">
</span><span class="c1"># =&gt; %{a: "foobar"}
</span><span class="w">
</span><span class="c1"># Map to a struct:
</span><span class="w"></span><span class="gp unselectable">iex&gt; </span><span class="n">struct</span><span class="p" data-group-id="7157529561-5">(</span><span class="nc">BestStructEver</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="7157529561-6">%{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="err">\</span><span class="err">"</span><span class="n">foobar</span><span class="err">\</span><span class="err">"</span><span class="p" data-group-id="7157529561-6">}</span><span class="p" data-group-id="7157529561-5">)</span><span class="w">
</span><span class="c1"># =&gt; %BestStructEver{a: "foobar"}
</span><span class="w">
</span><span class="c1"># Note: The struct function is from Kernel, so `Kernel.` can be omitted.</span></code></pre>

diff --git a/before b/after
index 55d1d44..37f2cc3 100644
--- a/before
+++ b/after
@@ -1,17 +1,13 @@
-<pre class="highlight"><code><span class="kd">defmodule</span><span class="w"> </span><span class="nc">BestStructEver</span><span class="w"> </span><span class="k" data-group-id="8518303302-1">do</span><span class="w">
-  </span><span class="kd">defstruct</span><span class="w"> </span><span class="p" data-group-id="8518303302-2">[</span><span class="ss">:a</span><span class="p" data-group-id="8518303302-2">]</span><span class="w">
-</span><span class="k" data-group-id="8518303302-1">end</span><span class="w">
+<pre class="highlight"><code><span class="kd">defmodule</span><span class="w"> </span><span class="nc">BestStructEver</span><span class="w"> </span><span class="k" data-group-id="7157529561-1">do</span><span class="w">
+  </span><span class="kd">defstruct</span><span class="w"> </span><span class="p" data-group-id="7157529561-2">[</span><span class="ss">:a</span><span class="p" data-group-id="7157529561-2">]</span><span class="w">
+</span><span class="k" data-group-id="7157529561-1">end</span><span class="w">
 
 </span><span class="c1"># Struct to simple map:
-</span><span class="w">
-</span><span class="gp unselectable">iex&gt; </span><span class="nc">Map</span><span class="o">.</span><span class="n">from_struct</span><span class="p" data-group-id="8518303302-3">(</span><span class="p" data-group-id="8518303302-4">%</span><span class="nc" data-group-id="8518303302-4">BestStructEver</span><span class="p" data-group-id="8518303302-4">{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="s">"foobar"</span><span class="p" data-group-id="8518303302-4">}</span><span class="p" data-group-id="8518303302-3">)</span><span class="w">
+</span><span class="w"></span><span class="gp unselectable">iex&gt; </span><span class="nc">Map</span><span class="o">.</span><span class="n">from_struct</span><span class="p" data-group-id="7157529561-3">(</span><span class="p" data-group-id="7157529561-4">%</span><span class="nc" data-group-id="7157529561-4">BestStructEver</span><span class="p" data-group-id="7157529561-4">{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="s">"foobar"</span><span class="p" data-group-id="7157529561-4">}</span><span class="p" data-group-id="7157529561-3">)</span><span class="w">
 </span><span class="c1"># =&gt; %{a: "foobar"}
 </span><span class="w">
-
 </span><span class="c1"># Map to a struct:
+</span><span class="w"></span><span class="gp unselectable">iex&gt; </span><span class="n">struct</span><span class="p" data-group-id="7157529561-5">(</span><span class="nc">BestStructEver</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="7157529561-6">%{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="err">\</span><span class="err">"</span><span class="n">foobar</span><span class="err">\</span><span class="err">"</span><span class="p" data-group-id="7157529561-6">}</span><span class="p" data-group-id="7157529561-5">)</span><span class="w">
+</span><span class="c1"># =&gt; %BestStructEver{a: "foobar"}
 </span><span class="w">
-</span><span class="gp unselectable">iex&gt; </span><span class="n">struct</span><span class="p" data-group-id="8518303302-5">(</span><span class="nc">BestStructEver</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8518303302-6">%{</span><span class="ss">a</span><span class="p">:</span><span class="w"> </span><span class="s">"foobar"</span><span class="p" data-group-id="8518303302-6">}</span><span class="p" data-group-id="8518303302-5">)</span><span class="w">
-</span><span class="c1"># =&gt; %BestStructEver{a: "foobar"}
-</span><span class="w">
-
 </span><span class="c1"># Note: The struct function is from Kernel, so `Kernel.` can be omitted.</span></code></pre>

The same result happens when I String.replace(code, "\r\n", "\n") before passing it to makeup, so I don't believe that's causing it yet.