TOC header anchor generator doesn't replace dots

[olivierlacan]

This is sort of a minor aesthetic issue, but generally I feel like any heading text being processed for a TOC header should follow the kind of conversion that ActiveSupport::Inflector#parameterize does since it's going to end up in a URL:

parameterize "## 1. Commit your code in your feature branch."
=> "1-commit-your-code-in-your-feature-branch"

However this is how Redcarpet currently handles the same string:

markdown = Redcarpet::Markdown.new(Redcarpet::Render::HTML_TOC, with_toc_data: true)
=> #<Redcarpet::Markdown:0x007fb013172d28 @renderer=#<Redcarpet::Render::HTML_TOC:0x007fb013172d50>>
markdown.render "## 1. Commit your code in your feature branch."
=> "<ul>\n<li>\n<a href=\"#1.-commit-your-code-in-your-feature-branch.\">1. Commit your code in your feature branch.</a>\n</li>\n</ul>\n"

Seems like a simple regex update inside of header_anchor would do the trick. It currently uses: <\\/?[^>]*>

This is the regex that parameterize uses: /[^a-z0-9\-_]+/i

C isn't my strong suit, but I can try to provide a PR for this if it seems sane.

[timcheadle]

I spent some time looking into this today, and it's a bit more complicated than it seems.

First off, header_anchor() currently only strips spaces. There are many more "unsafe" characters to strip from id= attributes and anchor links. I'm playing with this code as a replacement:

char *header_anchor(struct buf *text)
{
  VALUE str = rb_str_new2(bufcstr(text));
  VALUE tags_regex = rb_reg_new("<\\/?[^>]*>", 10 /* length */, 0);
  VALUE unsafe_char_regex = rb_reg_new("[ $&+,\\/:;=?@\"<>#%{}|\\^~\\[\\]`]+", 31, 0);
  VALUE trailing_dash_regex = rb_reg_new("-\\Z", 3, 0);

  VALUE heading = rb_funcall(str, rb_intern("gsub"), 2, tags_regex, rb_str_new2(""));
  heading = rb_funcall(heading, rb_intern("gsub"), 2, unsafe_char_regex, rb_str_new2("-"));
  heading = rb_funcall(heading, rb_intern("gsub"), 2, trailing_dash_regex, rb_str_new2(""));
  heading = rb_funcall(heading, rb_intern("downcase"), 0);

  return StringValueCStr(heading);
}

With this code, the TOC links render just fine. This markdown:

# How does "this" work?

Becomes this TOS link:

<ul>
  <li>
    <a href="#how-does-this-work">How does "this" work?</a>
  </li>
</ul>

The problem is in the regular header anchors. That same markdown renders as this header (with :with_toc_data passed in):

<h1 id="how-does-quot-this-quot-work">How does &quot;this&quot; work?</h1>

The reason for this is that the *text passed into header_anchor() for header tags has been run through escape_html() before it's run through the regular expressions to strip unsafe characters. Short of including the unescape utilities from Houdini, I'm not well versed enough with this code to fix that issue. Unescaping a string that you escaped earlier seems silly, anyway.

[robin850]

Hello there,

Thanks for your inputs on this! I agree that the TOC headers are a hot-spot that we should improve for the coming release. At least, you mentioned these different issues:

• Unescaping of non-alphanumeric or non-common (e.g. asterisks) characters
• Inconsistency between HTML and HTML_TOC for the header anchors

However, we'd love to rewrite this area of the code without the Ruby C API relying on regular regex.h (see 6d881e8) so there is a lot more work to do here. I've taken the liberty to snug the issue under the 3.2.0 milestone. I will try to address this as soon as possible unless someone wants to send a patch. :-)

[robin850]

Ok guys this should has been resolved by the aforementioned pull request ; thanks again for reporting ! :-)

[olivierlacan]

@robin850 Oh that's great, thanks! ❤️

[timcheadle]

Nice work! My only comment is that maybe the tests should include a double quote in one of the strings, but I have every confidence that it works regardless.