Adding more unicode characters to the generated *.tex file

[Paebbels]

The generated *.tex file already contains a section for additional Unicode characters like this:

\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
  \DeclareUnicodeCharacter{00A0}{\nobreakspace}
\fi

However, I need to add more characters, to build my documentation on my local machine. The build on ReadTheDocs seems to have no problems like this. I find it annoying to patch this 3.5 MiB file by hand to insert more characters before I launch the next translation step (starting pdflatex).

The characters are generated from <mmlalias.txt>.
This is my patched Unicode character list:

\usepackage[utf8]{inputenc}
\ifdefined\DeclareUnicodeCharacter
  \DeclareUnicodeCharacter{00A0}{\nobreakspace}
  \DeclareUnicodeCharacter{2265}{$\geq$}
  \DeclareUnicodeCharacter{21D2}{$\Rightarrow$}
\fi

I identified these characters by try-and-error debugging. More might follow so I'm looking for a longtime solution.

Questions:

• How can I automate this with Sphinx (in a way it runs on RTD too)?
• Is there a way to load more characters from an external file?
• Can Sphinx add the used substitutions automatically? (preferred)

[tk0miya]

How can I automate this with Sphinx (in a way it runs on RTD too)?

You can overwrite it through latex_elements['utf8extra'].
Now it is defined as following by default (in stable HEAD):

        'utf8extra':   ('\\ifdefined\\DeclareUnicodeCharacter\n'
                        ' \\ifdefined\\DeclareUnicodeCharacterAsOptional\\else\n'
                        '  \\DeclareUnicodeCharacter{00A0}{\\nobreakspace}\n'
                        '\\fi\\fi'),

Is there a way to load more characters from an external file?

Please write code in your conf.py.

Can Sphinx add the used substitutions automatically? (preferred)

IMO, it is better to use unicode-supported LaTeX engine (XeTeX or LuaTeX).
Supporting utf8 characters are very tired work.
refs: #3444

[Paebbels]

Is LuaTeX a replacement without side effect for pdflatex? I'm on Windows with a Miktex (2.9 x86_64) installation.

I'm using a translated Makefile as make.ps1 which triggers ... -b latex. luatex is not an option in the provided makefile.

[tk0miya]

Sorry, I don't know that because I'm not an expert of TeX.
And we Japanese usually uses pTeX series (which is specialized to Japanese processing).

[Paebbels]

OK. I have started LuaLaTeX, but it stops with unknown control sequence errors.

I also tried XeLaTeX that worked and generated the expected PDF file without adding more Unicode characters.

As this is nice to know, but non of the flows is supported on RTD. So I'll stick to manually adding characters to the preamble as propose in the first answer.

[jfbu]

Hi @Paebbels ! I am not familiar with it but there might be a pdflatex alternative which is utf8x option to inputenc.

latex_elements = {
   'inputenc' : '\\usepackage[utf8x]{inputenc}',
   'utf8extra': '',
}

It has capacity of handling on the fly Unicode characters, you can also use \DeclareUnicodeCharacter but beware the first argument which must either be a decimal number or if hexadecimal with an initial " (its syntax diverges from the \DeclareUnicodeCharacter of utf8 option).

\DeclareUnicodeCharacter{"21D2}{\ensuremath{\Rightarrow}}

In my brief testing I did not need explicit such declarations,

but from having looked briefly at utf8x doc, it may depend of circumstances where your Unicode chars first appear in document. Besides, on older TeX install maybe you need additional \\usepackage{ucs} as first thing in the inputenc key but on my install the utf8x option triggers that automatically.

The utf8x option was developed years before the LaTeX core provided similar functionality, in fact at that time it was called utf8 but LaTeX core later preempted the utf8 name, causing the renaming to utf8x. Then the LaTeX team made different choices and among them the syntax of \DeclareUnicodeCharacter.

[jfbu]

About using LuaTeX, this is option only if using always latest TeX install. Indeed, it evolved a lot over the last five years and only recently reached 1.0 status. And this will be in TeXLive 2017, the TeXLive 2016 (which generally does not update binaries during its yearly life span) still has earlier version, which is beta version. Many backwards incompatible changes happened over the years before 1.0. For example Sphinx pay attention to load package luatex85 else, all LuaTeX builds fail starting with TeXLive 2016 because that version dropped many \pdf... prefixed names which are expected by may older macro packages.

LuaTeX and XeTeX each has its problems with math. They are not 100% backwards compatible with pdfTeX in this respect, but it may not matter to most users. LuaTeX incorporates Lua scripting language and hooks into the core TeX hence it allows things completely impossible at macro level in TeX.

LuaTeX and XeTeX are best used with OpenType fonts, although they may be used with classic TeX fonts for backwards compatibility (very good for text, less good for math).

Regarding Sphinx, currently on master HEAD, they will load package fontspec. This will make Latin Modern OpenType the default font, with its Math counterpart. But fontspec did not yet support LuaTeX in 2009, so this change in Sphinx was made possible from decision #3070 to support Ubuntu/trusty for 1.6. Anyway, as said above, anyone using LuaTeX must make sure its TeX install is always the latest one.

[Paebbels]

I have an up-to-date Miktex installation. LuaLaTeX reports:

This is LuaTeX, Version 1.0.0 (MiKTeX 2.9.6210 64-bit)

[jfbu]

Thus, you are all clear for LuaLaTeX use ! but did you use LuaLaTeX via latex_engine = 'lualatex' in the conf.py or manually on the tex file produced by Sphinx without such config ? in the latter case you would need to add \usepackage{luatex85} to the tex file preamble.