slugify replaces Tamil vowel marks with hyphen

[deepestblue]

• I updated to the latest github-pages
• I ran bundle exec jekyll doctor to check my configuration
• I read the contributing document at https://jekyllrb.com/docs/contributing/

My Environment

Software	Version(s)
Operating System	OSX 10.15.2 (19C57)
jekyll	3.8.5
github-pages	203 (Latest)

---

Expected Behaviour

I had a post with a Unicode Tamil file-name (மல்லிப்பூ வகைகள்.md). I expected the generated permalink to include all the letters and combining marks, while converting spaces to hyphens. Specifically, I expected the generated file-name to be மல்லிப்பூ-வகைகள்.html.

Current Behaviour

Instead, it generated something like மல-லிப-பூ-வகைகள.html, breaking up words in unexpected places.

I looked at the code, and in jekyll-3.8.5/lib/jekyll/utils.rb, I found replace_character_sequence_with_hyphen which replaces /[^[:alnum:]._~!$&'()+,;=@]+/ with -. The problem is that there many Unicode letter/mark codepoints that don't match alnum: in my specific case, U+0BCD. It turns out that the Alphabetic Unicode property only applies to some of what we'd want to consider Letters/Marks.

Instead, the recommendation I got on SO was to use Unicode categories. My suggested fix would be to change the Regex to match /[^\p{M}\p{L}\p{Nd}._~!$&'()+,;=@]+/.

[ashmaroli]

Hi @deepestblue
Since you've a possible fix in mind, feel free to open a pull request with your use-case as a test-scenario.

On a side-note, even if the patch were to get accepted, it may not get backported to Jekyll 3.8 (the current version is Jekyll 4.0.0)

[deepestblue]

@ashmaroli I don't have jekyll 4.0.0, and not being a Ruby expert, am not comfortable trying to parallel install both 4.0.0 and 3.8.5 :-( I'm guessing it's not useful to send out a patch against 3.8.5.

[ashmaroli]

Clarifications:

• GitHub Pages locks to a particular version of jekyll, plugins and themes. It is currently on Jekyll 3.8.5.
  The last release in 3.8 series was v3.8.6. Even if we ship a v3.8.7, the chances of GHP updating to Jekyll-3.8.7 immediately is very low.
• FWIW, this can't be considered as a bug-fix because it will cause URL changes to existing ones generated from other non-latin languages. So, this has to be an opt-in feature.
• If you're not comfortable with Ruby and the ecosystem, no issues. I can always mark this as up-for-grabs for someone looking to contribute to this project. It will be a long-hanging fruit for them since you've already suggested the solution.

[swanson]

👋 @josh-works and I are going to take a stab at this in the next week or so

[ashmaroli]

Resolved via #8167