sagemathgh-36861: Use lualatex for better unicode support

    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

to support sagemath#36857.

There have been numerous problems due to limited support of unicode in
pdflatex. I see no reason why we should not switch to modern tex engine
lualatex, as suggested in

https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-
latex_engine

We set lualatex as the default engine for
- building the sage documentation, in `sage_docbuild/`
- rendering pdf images, in `src/sage/misc/latex.py`

In  `src/sage/misc/latex.py`, the default engine is determined by
availability of the engines  with the order of preference: lualatex,
xelatex, pdflatex.

Along the way, we
- remove latex-related code deprecated in sagemath#32650.
- make lots of cosmetic edits in `src/sage/misc/latex.py`
- add support "lualatex" as an alternative to "[pdf|xe]latex" in
`src/sage/misc/latex.py`

We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy`
as new dependencies of `sagemath_doc_pdf` in addition to `texlive`
optional package. It seems that depending on the version and the
platform, `texlive` may already include the new dependencies. If not,
you need to install the new dependencies (as `_recommended` by the dummy
package)

On Ubuntu, the requirement is fulfilled by
```
sudo apt install texlive texlive-luatex fonts-freefont-otf xindy
```

The built pdf doc is available here: https://deploy-livedoc--
sagemath.netlify.app

To test `view` via lualatex, for example,
```
latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof
ont{Arial}")
view('Εύρηκα')
```
you should build sage on your local platform with this PR. You may need
to install `texlive-full` before testing. It seems that `texlive` (2019)
is not enough even for the `view` via pdftex.

sage-devel thread seeking Unicode testers:
https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I


<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
Resolves sagemath#18370
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36861
Reported by: Kwankyu Lee
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe

.github/workflows/doc-build-pdf.yml

@@ -57,7 +57,7 @@ jobs:
           export PATH="build/bin:$PATH"
           eval $(sage-print-system-package-command auto update)
           eval $(sage-print-system-package-command auto --yes --no-install-recommends install zip)
-          eval $(sage-print-system-package-command auto --spkg --yes --no-install-recommends install git texlive)
+          eval $(sage-print-system-package-command auto --spkg --yes --no-install-recommends install git texlive texlive_luatex free_fonts xindy)
 
       - name: Add prebuilt tree as a worktree
         id: worktree

.github/workflows/doc-build.yml

@@ -161,7 +161,7 @@ jobs:
           export PATH="build/bin:$PATH"
           eval $(sage-print-system-package-command auto update)
           eval $(sage-print-system-package-command auto --yes --no-install-recommends install zip)
-          eval $(sage-print-system-package-command auto --spkg --yes --no-install-recommends install git texlive)
+          eval $(sage-print-system-package-command auto --spkg --yes --no-install-recommends install git texlive texlive_luatex free_fonts xindy)
           export SAGE_USE_CDNS=yes
           export SAGE_LIVE_DOC=yes
           export SAGE_JUPYTER_SERVER=binder:sagemath/sage-binder-env/dev

build/pkgs/_recommended/dependencies

@@ -1 +1 @@
-pandoc ffmpeg imagemagick texlive git libjpeg
+pandoc ffmpeg imagemagick git libjpeg texlive texlive_luatex free_fonts xindy

build/pkgs/free_fonts/SPKG.rst

@@ -0,0 +1,23 @@
+free_fonts: a free family of scalable outline fonts
+===================================================
+
+Description
+-----------
+
+This dummy package represents the GNU free fonts: a free family of scalable
+outline fonts, suitable for general use on computers and for desktop
+publishing. It is Unicode-encoded for compatibility with all modern operating
+systems.
+
+We do not have an SPKG for it. The purpose of this dummy package is to
+associate system package lists with it.
+
+License
+-------
+
+GNU General Public License GPLv3+
+
+Upstream Contact
+----------------
+
+https://www.gnu.org/software/freefont/

build/pkgs/free_fonts/distros/alpine.txt

@@ -0,0 +1 @@
+ttf-freefont

build/pkgs/free_fonts/distros/arch.txt

@@ -0,0 +1 @@
+gnu-free-fonts

build/pkgs/free_fonts/distros/conda.txt

@@ -0,0 +1 @@
+open-fonts

build/pkgs/free_fonts/distros/debian.txt

@@ -0,0 +1 @@
+fonts-freefont-otf

build/pkgs/free_fonts/distros/fedora.txt

@@ -0,0 +1 @@
+gnu-free-fonts

build/pkgs/free_fonts/distros/freebsd.txt

@@ -0,0 +1 @@
+x11-fonts/freefont-ttf

build/pkgs/free_fonts/distros/gentoo.txt

@@ -0,0 +1 @@
+media-fonts/freefont

build/pkgs/free_fonts/distros/macports.txt

@@ -0,0 +1 @@
+freefont-ttf

build/pkgs/free_fonts/distros/opensuse.txt

@@ -0,0 +1 @@
+gnu-free-fonts

build/pkgs/free_fonts/distros/repology.txt

@@ -0,0 +1 @@
+gnu-freefont

build/pkgs/free_fonts/distros/void.txt

@@ -0,0 +1 @@
+freefont-ttf

build/pkgs/free_fonts/spkg-configure.m4

@@ -0,0 +1,16 @@
+SAGE_SPKG_CONFIGURE([free_fonts], [
+    sage_spkg_install_free_fonts=yes
+    m4_foreach([font],
+               [FreeSerif.ttf,FreeSerif.otf],
+        [
+        AC_MSG_CHECKING([for ]font)
+        AS_IF([kpsewhich ]font[ >& AS_MESSAGE_LOG_FD 2>&1], [
+            AC_MSG_RESULT([yes])
+            sage_spkg_install_free_fonts=no
+            break
+        ], [
+            AC_MSG_RESULT([no])
+        ])
+    ])
+])
+

build/pkgs/free_fonts/type

@@ -0,0 +1 @@
+optional

build/pkgs/sagemath_doc_pdf/dependencies

@@ -1 +1 @@
-sagemath_doc_html texlive
+sagemath_doc_html texlive texlive_luatex free_fonts xindy

build/pkgs/texlive_luatex/SPKG.rst

@@ -0,0 +1,21 @@
+texlive_luatex: LuaTeX packages
+===============================
+
+Description
+-----------
+
+Packages for LuaTeX, a TeX engine using Lua as an embedded scripting and
+extension language, with native support for Unicode, OpenType/TrueType fonts,
+and both PDF and DVI output.
+
+The purpose of this dummy package is to associate system package lists with it.
+
+License
+-------
+
+GNU General Public License version 2.0 (GPLv2)
+
+Upstream Contact
+----------------
+
+https://www.luatex.org/

build/pkgs/texlive_luatex/dependencies

@@ -0,0 +1 @@
+texlive

build/pkgs/texlive_luatex/distros/alpine.txt

@@ -0,0 +1 @@
+texlive-luatex

build/pkgs/texlive_luatex/distros/arch.txt

@@ -0,0 +1 @@
+texlive-collection-luatex

build/pkgs/texlive_luatex/distros/cygwin.txt

@@ -0,0 +1 @@
+texlive

build/pkgs/texlive_luatex/distros/debian.txt

@@ -0,0 +1 @@
+texlive-luatex

build/pkgs/texlive_luatex/distros/fedora.txt

@@ -0,0 +1 @@
+texlive-luatex

build/pkgs/texlive_luatex/distros/gentoo.txt

@@ -0,0 +1 @@
+dev-texlive/texlive-luatex

build/pkgs/texlive_luatex/distros/macports.txt

@@ -0,0 +1 @@
+texlive-luatex

build/pkgs/texlive_luatex/distros/opensuse.txt

@@ -0,0 +1 @@
+texlive-luatex

build/pkgs/texlive_luatex/distros/repology.txt

@@ -0,0 +1 @@
+texlive-luatex

build/pkgs/texlive_luatex/spkg-configure.m4

@@ -0,0 +1,10 @@
+SAGE_SPKG_CONFIGURE([texlive_luatex], [
+    sage_spkg_install_texlive_luatex=no
+    AC_MSG_CHECKING([for luaotfload-main.lua])
+    AS_IF([kpsewhich luaotfload-main.lua >& AS_MESSAGE_LOG_FD 2>&1], [
+        AC_MSG_RESULT([yes])
+    ], [
+        AC_MSG_RESULT([no])
+        sage_spkg_install_texlive_luatex=yes
+    ])
+])

build/pkgs/texlive_luatex/type

@@ -0,0 +1 @@
+optional

build/pkgs/xindy/SPKG.rst

@@ -0,0 +1,20 @@
+xindy: a general-purpose index processor
+========================================
+
+Description
+-----------
+
+This dummy package represents xindy: xindy was developed after an impasse had been encountered in the attempt to complete internationalisation of makeindex. Xindy can be used to process indexes for documents marked up using (LA)TEX, Nroff family and SGML-based languages. Xindy is highly configurable, both in markup terms and in terms of the collating order of the text being processed.
+
+We do not have an SPKG for it. The purpose of this dummy package is to
+associate system package lists with it.
+
+License
+-------
+
+GNU General Public License version 2.0 (GPLv2)
+
+Upstream Contact
+----------------
+
+http://www.xindy.org/

build/pkgs/xindy/distros/debian.txt

@@ -0,0 +1 @@
+xindy

build/pkgs/xindy/distros/macports.txt

@@ -0,0 +1 @@
+xindy

build/pkgs/xindy/distros/opensuse.txt

@@ -0,0 +1 @@
+xindy

build/pkgs/xindy/distros/repology.txt

@@ -0,0 +1 @@
+xindy

build/pkgs/xindy/spkg-configure.m4

@@ -0,0 +1,5 @@
+SAGE_SPKG_CONFIGURE([xindy], [
+    sage_spkg_install_xindy=no
+    AC_PATH_PROG([XINDY], [xindy])
+    AS_IF([test -z "$XINDY"], [sage_spkg_install_xindy=yes])
+])

build/pkgs/xindy/type

@@ -0,0 +1 @@
+optional

src/doc/fr/tutorial/conf.py

@@ -39,9 +39,6 @@
    'The Sage Group', 'manual'),
 ]
 
-# Additional LaTeX stuff for the French version
-#latex_elements['preamble'] += '\\DeclareUnicodeCharacter{00A0}{\\nobreakspace}\n'
-
 # the definition of \\at in the standard preamble of the sphinx doc
 # conflicts with that in babel/french[b]
 latex_elements['preamble'] += '\\let\\at\\undefined'

src/doc/ja/a_tour_of_sage/conf.py

@@ -25,6 +25,10 @@
 name = 'a_tour_of_sage'
 language = "ja"
 
+# The LaTeX engine to build the docs in Japanese.
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine
+latex_engine = 'uplatex'
+
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".
 html_title = project + " v" + release

src/doc/ja/tutorial/conf.py

@@ -25,6 +25,10 @@
 name = 'tutorial-jp'
 language = "ja"
 
+# The LaTeX engine to build the docs in Japanese.
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine
+latex_engine = 'uplatex'
+
 # The name for this set of Sphinx documents.  If None, it defaults to
 # "<project> v<release> documentation".
 html_title = project + " v" + release

src/doc/ru/tutorial/conf.py

@@ -38,6 +38,3 @@
   ('index', 'SageTutorial_ru.tex', 'Sage Tutorial in Russian',
    'The Sage Development Team', 'manual'),
 ]
-
-# Additional LaTeX stuff if necessary:
-#latex_elements['preamble'] += '\\DeclareUnicodeCharacter{00A0}{\\nobreakspace}\n'