-
-
Notifications
You must be signed in to change notification settings - Fork 473
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use lualatex for better unicode support #36861
Conversation
AFAIK, the official successor of pfd(la)tex is lua(la)tex, not xe(la)tex. See e.g. https://tex.stackexchange.com/questions/126206/why-choose-lualatex-over-xelatex and https://en.wikipedia.org/wiki/LuaTeX It appears that xetex is not developed further since 2019 or so. |
There's no "official" successor.
Like TeX itself, it is not a flaw that xetex is not further developed. It is a great flaw if it was not maintained. I have (and had) no problem at all with xelatex in typesetting my scripts in English and Korean with little bits of Chinese and Greek. I'd say xetex is just stable. I have zero experience with LuaTeX. I know it is famous because of its scripting capability with Lua language. Do we need the capability? I just hope that with minimal efforts (on my side), xelatex solves the problems we now face. If you want to solve more problems with more capable luatex, I welcome you opening a PR as a sequel. Then this PR will serve as a stepping stone. |
luatex is at least the official successor of pdftex in TexLive: https://en.wikipedia.org/wiki/LuaTeX I use lualatex, it's quite nice, and its unicode support is better in some ways. Also, Why do we need two PRs, and why choose an older project which is being phased out in favour of a newer one? |
OK. Good.
Then would you open a PR and work on it? I will happily close this PR then. (because I do not want to spend time in switching from xelatex to luatex now) |
OK, let me try lualatex - it it's a bit more work, as Sage does know about xelatex, but does not know about lualatex. |
by the way, I tested this PR, and saw problems with Japanese and Cyrillic. So it's one way or another, more work needed, it seems. |
Thanks. This PR is in testing phase (not ready for review yet). I will work on the problems. You may want to prepare your luatex PR as a sequel to this PR, to not duplicate efforts. |
I don't recognize any problem from the built pdf docs. What problems did you see? Are they new? |
Ah sorry, my pdf docs were built by pdflatex, somehow... |
I added support of lualatex in while setting xelatex as the default engine for rendering pdfs, in If this setup works out well, then later, in your PR, you may replace xelatex with lualatex in the places setting the default engine. |
On Tue, Dec 12, 2023 at 01:33:51AM -0800, Kwankyu Lee wrote:
8a966dd
while setting xelatex as the default engine for rendering pdfs, in `src/sage/misc/latex.py`.
one needs to change build/pkgs/texlive/spkg-configure.m4 too. The list
of LaTeX packages,
e.g. [unicode-math](https://ctan.org/pkg/unicode-math?lang=en) is most probably a good idea,
and the programs (No need for dvi* things any more) required there should be adjusted.
|
That may be a useful addition, though it is not required.
We support [pdf|xe|lua]latex in addition to latex. dvi* things are still necessary for latex. Anyway, that is a separate concern. |
From Sphinx doc,
How should we proceed? Perhaps We need a dummy package created from https://repology.org/project/fonts:gnu-freefont/versions @mkoeppe any tip? |
Why do we need to support I discussed pdftex/xetex/luatex compatibility with David Carlisle here: https://tex.stackexchange.com/questions/601579/unicode-maths-in-pdflatex - you can see various details there |
|
The benefit of not using latex/pdflatex is that you can use unicode freely, without worrying about missing |
By the way, "unicode in maths mode" is not in the scope of this PR. |
it is very much in the scope, as using xelatex will easily break compatibility with (pdf)latex. To ensure the compatibility is not broken, you'd need to test the non-default (pdf)latex settings. |
If a user chooses latex/pdflatex, he simply has chosen to not use unicode freely. We don't have to support him as much as xelatex/lualatex users regarding unicode. Don't worry about missing |
So I submit a PR with unicode math, and it passes the default (say, xelatex) pdf docs builder. Not testing the PR with non-default builder means that our docs might be broken in sense they can't be built with (pdf)latex any more. |
|
In addition, I may write some guide later in the release tour. Would that be enough? |
No. |
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> to support sagemath#36857. There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in https://www.sphinx-doc.org/en/master/usage/configuration.html#confval- latex_engine We set lualatex as the default engine for - building the sage documentation, in `sage_docbuild/` - rendering pdf images, in `src/sage/misc/latex.py` In `src/sage/misc/latex.py`, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex. Along the way, we - remove latex-related code deprecated in sagemath#32650. - make lots of cosmetic edits in `src/sage/misc/latex.py` - add support "lualatex" as an alternative to "[pdf|xe]latex" in `src/sage/misc/latex.py` We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy` as new dependencies of `sagemath_doc_pdf` in addition to `texlive` optional package. It seems that depending on the version and the platform, `texlive` may already include the new dependencies. If not, you need to install the new dependencies (as `_recommended` by the dummy package) On Ubuntu, the requirement is fulfilled by ``` sudo apt install texlive texlive-luatex fonts-freefont-otf xindy ``` The built pdf doc is available here: https://deploy-livedoc-- sagemath.netlify.app To test `view` via lualatex, for example, ``` latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof ont{Arial}") view('Εύρηκα') ``` you should build sage on your local platform with this PR. You may need to install `texlive-full` before testing. It seems that `texlive` (2019) is not enough even for the `view` via pdftex. sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36861 Reported by: Kwankyu Lee Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
I am currently getting one error in building the pdf doc with this
Am I missing some package? |
is it with lualatex ? |
Most definitely lualatex. |
do you see this warning in CI logs too? I.e., is the error consequence of the warning? might potentially be some locale nonsense... |
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> to support sagemath#36857. There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in https://www.sphinx-doc.org/en/master/usage/configuration.html#confval- latex_engine We set lualatex as the default engine for - building the sage documentation, in `sage_docbuild/` - rendering pdf images, in `src/sage/misc/latex.py` In `src/sage/misc/latex.py`, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex. Along the way, we - remove latex-related code deprecated in sagemath#32650. - make lots of cosmetic edits in `src/sage/misc/latex.py` - add support "lualatex" as an alternative to "[pdf|xe]latex" in `src/sage/misc/latex.py` We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy` as new dependencies of `sagemath_doc_pdf` in addition to `texlive` optional package. It seems that depending on the version and the platform, `texlive` may already include the new dependencies. If not, you need to install the new dependencies (as `_recommended` by the dummy package) On Ubuntu, the requirement is fulfilled by ``` sudo apt install texlive texlive-luatex fonts-freefont-otf xindy ``` The built pdf doc is available here: https://deploy-livedoc-- sagemath.netlify.app To test `view` via lualatex, for example, ``` latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof ont{Arial}") view('Εύρηκα') ``` you should build sage on your local platform with this PR. You may need to install `texlive-full` before testing. It seems that `texlive` (2019) is not enough even for the `view` via pdftex. sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> Resolves sagemath#18370 <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36861 Reported by: Kwankyu Lee Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
I see some of the warnings in the CI, with slightly different line number. Although I do not see the |
maybe you can post various .log files? |
A bit of the folder where things fails. Includes the full |
Sorry for not responding. I am out of town and come back after holidays.
…On Sun, Dec 24, 2023 at 7:15 AM François Bissey ***@***.***> wrote:
A bit of the folder where things fails. Includes the full coding.log and
also coding.tex for those who want to inspect it.
http://202.36.178.9/sage/coding.tar.xz
—
Reply to this email directly, view it on GitHub
<#36861 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AALDPZICWTUJB3CJCVI6XKDYK5JYTAVCNFSM6AAAAABAPWXSCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGM3TSMZZGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> to support sagemath#36857. There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in https://www.sphinx-doc.org/en/master/usage/configuration.html#confval- latex_engine We set lualatex as the default engine for - building the sage documentation, in `sage_docbuild/` - rendering pdf images, in `src/sage/misc/latex.py` In `src/sage/misc/latex.py`, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex. Along the way, we - remove latex-related code deprecated in sagemath#32650. - make lots of cosmetic edits in `src/sage/misc/latex.py` - add support "lualatex" as an alternative to "[pdf|xe]latex" in `src/sage/misc/latex.py` We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy` as new dependencies of `sagemath_doc_pdf` in addition to `texlive` optional package. It seems that depending on the version and the platform, `texlive` may already include the new dependencies. If not, you need to install the new dependencies (as `_recommended` by the dummy package) On Ubuntu, the requirement is fulfilled by ``` sudo apt install texlive texlive-luatex fonts-freefont-otf xindy ``` The built pdf doc is available here: https://deploy-livedoc-- sagemath.netlify.app To test `view` via lualatex, for example, ``` latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof ont{Arial}") view('Εύρηκα') ``` you should build sage on your local platform with this PR. You may need to install `texlive-full` before testing. It seems that `texlive` (2019) is not enough even for the `view` via pdftex. sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> Resolves sagemath#18370 <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [x] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36861 Reported by: Kwankyu Lee Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
I don't know. I don't see any hint from your logs about what went wrong. If you still have it, I would just recommend to upgrade your texlive or reinstall gnu free fonts and wish a good luck. |
Yes, in the end, texlive from the gentoo main tree proved too old. Moving to texlive 2023 fixed the issue. |
Good to hear that. texlive 2023 seems to also include |
I was on texlive 2021 for info. |
Now that I see this, out of curiosity, doesn't this slow down all LaTeX compilations a lot? LuLaTeX is much slower than pdflatex. |
That was noted here: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I/m/TlMGEkCwAAAJ |
lualatex, in its default configuration, is slower to start up. There are, however, various ways to speed it up. All these need extra investigation. The big plus here is that we finally sorted our Unicode troubles. |
This is using unicode box-drawing characters to make better-looking tables. ### 📝 Checklist - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. ### ⌛ Dependencies - sagemath#36861 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> URL: sagemath#36857 Reported by: Frédéric Chapoton Reviewer(s): Kwankyu Lee
This is using unicode box-drawing characters to make better-looking tables. ### 📝 Checklist - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. ### ⌛ Dependencies - sagemath#36861 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> URL: sagemath#36857 Reported by: Frédéric Chapoton Reviewer(s): Kwankyu Lee
This is using unicode box-drawing characters to make better-looking tables. ### 📝 Checklist - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. ### ⌛ Dependencies - sagemath#36861 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> URL: sagemath#36857 Reported by: Frédéric Chapoton Reviewer(s): Kwankyu Lee
to support #36857.
There have been numerous problems due to limited support of unicode in pdflatex. I see no reason why we should not switch to modern tex engine lualatex, as suggested in
https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine
We set lualatex as the default engine for
sage_docbuild/
src/sage/misc/latex.py
In
src/sage/misc/latex.py
, the default engine is determined by availability of the engines with the order of preference: lualatex, xelatex, pdflatex.Along the way, we
src/sage/misc/latex.py
src/sage/misc/latex.py
We add optional (dummy) packages
texlive_luatex
,free_fonts
,xindy
as new dependencies ofsagemath_doc_pdf
in addition totexlive
optional package. It seems that depending on the version and the platform,texlive
may already include the new dependencies. If not, you need to install the new dependencies (as_recommended
by the dummy package)On Ubuntu, the requirement is fulfilled by
The built pdf doc is available here: https://deploy-livedoc--sagemath.netlify.app
To test
view
via lualatex, for example,you should build sage on your local platform with this PR. You may need to install
texlive-full
before testing. It seems thattexlive
(2019) is not enough even for theview
via pdftex.sage-devel thread seeking Unicode testers: https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I
Resolves #18370
📝 Checklist
⌛ Dependencies