Convert common TeX to unicode

[simonbyrne]

I set it up here, and noticed a few TeX artifacts.

I would suggest at least converting two dashes (--) to an en-dash (–) as those are very common and hard to type.

[ali-ramadhan]

PR #17 starts working on this (see below). I'll tag v0.1.1 or v0.2.0 shortly after it's merged.

@simonbyrne Do you know of a list of these kinds of replacements or should we just add to it as we go along?

I was able to find lists for math TeX to unicode (https://github.com/svenkreiss/unicodeit/blob/master/unicodeit/data.py) but not so much for text replacements.

[simonbyrne]

I agree we probably don't want the full unicodeit list, as it seems to include both math and text commands

[charleskawczynski]

Reopening this. We're still seeing issues, e.g. Jo { \~a } o Teixeira, over at the ClimateMachine.jl refs.

[simonbyrne]

This is due to spurious spaces being inserted by the BibTeX parser. Upstream issue is Humans-of-Julia/BibParser.jl#5.

[charleskawczynski]

Should we leave this open until the upstream is closed?

[simonbyrne]

Yes, probably a good idea.

[Azzaare]

Hi there! Sorry for the long wait, spurious braces should not be a problem anymore.

It might only be a crude parser that I handcrafted, but BibParser.jl got updated today (v0.1.11)
(the new parser should handle any valid BibTeX entry, but do not replace LaTeX commands from a @preamble nor converts LaTeX to Unicode)

[Azzaare]

I've created a GitHub repo to convert LaTeX ⇋ Unicode: https://github.com/Humans-of-Julia/LaTeXUniCode.jl
It is almost empty at the moment, but I will work on it during summer (as I will be in between two jobs, I can have some fun!)

Anyway, if some of you want to join, you're all welcome aboard.

[fingolfin]

The function tex2unicode is there, but it does not seem to be applied to pages, which is where I see them most often:

pages = {1 -- 45},

Could this be done?

[LazyScholar]

tex2unicoe is currently only applied to title?

DocumenterCitations.jl/src/bibliography.jl

Lines 50 to 54 in 886bbb7

	authors = xnames(entry)
	year = xyear(entry)
	link = xlink(entry)
	title = xtitle(entry) |> tex2unicode
	published_in = xin(entry)

Or am i misunderstanding it?

[fingolfin]

No you are right. Hmm, I thought I'd made a PR also applying it to the output of xin... guess I forgot :/

[LazyScholar]

I converted all my .bib files to Unicode therefore i did nor realize that applying it to the authors and maybe published_in might fix it for others.

@fingolfin do you want to make the PR (you can delete line 51 as with your last change the year is not needed any more)?

[LazyScholar]

Reopening this. We're still seeing issues, e.g. Jo { \~a } o Teixeira, over at the ClimateMachine.jl refs.

@charleskawczynski Is { \~a } valid Tex? As far as i know in order to get ã one have to use \~{a} or even \tilde{a} (not sure if the later one is supported by bibtex).
Source: https://en.wikibooks.org/wiki/LaTeX/Special_Characters#Escaped_codes