Unicode support in reference manual and other docs, e.g. developer manual #18370
Labels
Milestone
Comments
This comment has been minimized.
This comment has been minimized.
|
Changed keywords from none to unicode |
|
comment:3
xelatex can be used with sphinx >= 1.5 (#22252) |
|
comment:4
any progress here ? |
|
It's also needed in the developer manual. More and more tools output Unicode nowadays, including emoji chars, so one has to manually remove Unicode, as had to be done on #36117, for instance. |
dimpase
changed the title
2 tasks
vbraun
pushed a commit
to vbraun/sage
that referenced
this issue
Dec 23, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> to support sagemath#36857. There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in https://www.sphinx-doc.org/en/master/usage/configuration.html#confval- latex_engine We set lualatex as the default engine for - building the sage documentation, in `sage_docbuild/` - rendering pdf images, in `src/sage/misc/latex.py` In `src/sage/misc/latex.py`, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex. Along the way, we - remove latex-related code deprecated in sagemath#32650. - make lots of cosmetic edits in `src/sage/misc/latex.py` - add support "lualatex" as an alternative to "[pdf|xe]latex" in `src/sage/misc/latex.py` We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy` as new dependencies of `sagemath_doc_pdf` in addition to `texlive` optional package. It seems that depending on the version and the platform, `texlive` may already include the new dependencies. If not, you need to install the new dependencies (as `_recommended` by the dummy package) On Ubuntu, the requirement is fulfilled by ``` sudo apt install texlive texlive-luatex fonts-freefont-otf xindy ``` The built pdf doc is available here: https://deploy-livedoc-- sagemath.netlify.app To test `view` via lualatex, for example, ``` latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof ont{Arial}") view('Εύρηκα') ``` you should build sage on your local platform with this PR. You may need to install `texlive-full` before testing. It seems that `texlive` (2019) is not enough even for the `view` via pdftex. sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> Resolves sagemath#18370 <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36861 Reported by: Kwankyu Lee Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
vbraun
pushed a commit
to vbraun/sage
that referenced
this issue
Dec 24, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> to support sagemath#36857. There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in https://www.sphinx-doc.org/en/master/usage/configuration.html#confval- latex_engine We set lualatex as the default engine for - building the sage documentation, in `sage_docbuild/` - rendering pdf images, in `src/sage/misc/latex.py` In `src/sage/misc/latex.py`, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex. Along the way, we - remove latex-related code deprecated in sagemath#32650. - make lots of cosmetic edits in `src/sage/misc/latex.py` - add support "lualatex" as an alternative to "[pdf|xe]latex" in `src/sage/misc/latex.py` We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy` as new dependencies of `sagemath_doc_pdf` in addition to `texlive` optional package. It seems that depending on the version and the platform, `texlive` may already include the new dependencies. If not, you need to install the new dependencies (as `_recommended` by the dummy package) On Ubuntu, the requirement is fulfilled by ``` sudo apt install texlive texlive-luatex fonts-freefont-otf xindy ``` The built pdf doc is available here: https://deploy-livedoc-- sagemath.netlify.app To test `view` via lualatex, for example, ``` latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof ont{Arial}") view('Εύρηκα') ``` you should build sage on your local platform with this PR. You may need to install `texlive-full` before testing. It seems that `texlive` (2019) is not enough even for the `view` via pdftex. sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> Resolves sagemath#18370 <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36861 Reported by: Kwankyu Lee Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
vbraun
pushed a commit
to vbraun/sage
that referenced
this issue
Dec 26, 2023
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> to support sagemath#36857. There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in https://www.sphinx-doc.org/en/master/usage/configuration.html#confval- latex_engine We set lualatex as the default engine for - building the sage documentation, in `sage_docbuild/` - rendering pdf images, in `src/sage/misc/latex.py` In `src/sage/misc/latex.py`, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex. Along the way, we - remove latex-related code deprecated in sagemath#32650. - make lots of cosmetic edits in `src/sage/misc/latex.py` - add support "lualatex" as an alternative to "[pdf|xe]latex" in `src/sage/misc/latex.py` We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy` as new dependencies of `sagemath_doc_pdf` in addition to `texlive` optional package. It seems that depending on the version and the platform, `texlive` may already include the new dependencies. If not, you need to install the new dependencies (as `_recommended` by the dummy package) On Ubuntu, the requirement is fulfilled by ``` sudo apt install texlive texlive-luatex fonts-freefont-otf xindy ``` The built pdf doc is available here: https://deploy-livedoc-- sagemath.netlify.app To test `view` via lualatex, for example, ``` latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof ont{Arial}") view('Εύρηκα') ``` you should build sage on your local platform with this PR. You may need to install `texlive-full` before testing. It seems that `texlive` (2019) is not enough even for the `view` via pdftex. sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> Resolves sagemath#18370 <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36861 Reported by: Kwankyu Lee Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
LaTeX isn't particularly good at typesetting Unicode text. While the utf8 input encoding can help recognizing the most common symbols even if the input is Unicode, the output is essentially restricted to 8-bit encodings so every special symbol has to be set up somewhere.
src/doc/common/conf.pyhas a list of such setup rules, although some are pretty hackish, replacing unicode symbols with stuff that only looks remotely similar.Unicode in doctest output (and perhaps other parts of the reference manual as well) has caused problems in the past, e.g. ticket:15802 and #18270 comment:18. It is likely to cause more problems in the future, when we make more extensive use of Unicode in Sage output once #18357 gets implemented. So Unicode in the doctests is going to be increasingly common, and it would be nice if we didn't have to jump through hoops to get that represented in the PDF output.
As far as I understand it, both XeTeX /
xelatexand luaTeX /lualatexprovide a more modern approach to font handling, and considerably better Unicode support. I haven't yet gotten things to work that way, but my hope would be that we could gain access to all the symbols of a sufficiently complete Unicode font. Apparently, current versions of Sphinx don't officially support either of these engines yet. And things don't work out of the box. But perhaps things can be made to work with a bit of patching. If so, that might greatly improve the situation.See also #14153
CC: @nexttime @embray @jdemeyer @hivert
Component: documentation
Keywords: unicode
Issue created by migration from https://trac.sagemath.org/ticket/18370
The text was updated successfully, but these errors were encountered: