Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode support in reference manual and other docs, e.g. developer manual #18370

Closed
gagern mannequin opened this issue May 5, 2015 · 5 comments · Fixed by #36861
Closed

Unicode support in reference manual and other docs, e.g. developer manual #18370

gagern mannequin opened this issue May 5, 2015 · 5 comments · Fixed by #36861

Comments

@gagern
Copy link
Mannequin

gagern mannequin commented May 5, 2015

LaTeX isn't particularly good at typesetting Unicode text. While the utf8 input encoding can help recognizing the most common symbols even if the input is Unicode, the output is essentially restricted to 8-bit encodings so every special symbol has to be set up somewhere. src/doc/common/conf.py has a list of such setup rules, although some are pretty hackish, replacing unicode symbols with stuff that only looks remotely similar.

Unicode in doctest output (and perhaps other parts of the reference manual as well) has caused problems in the past, e.g. ticket:15802 and #18270 comment:18. It is likely to cause more problems in the future, when we make more extensive use of Unicode in Sage output once #18357 gets implemented. So Unicode in the doctests is going to be increasingly common, and it would be nice if we didn't have to jump through hoops to get that represented in the PDF output.

As far as I understand it, both XeTeX / xelatex and luaTeX / lualatex provide a more modern approach to font handling, and considerably better Unicode support. I haven't yet gotten things to work that way, but my hope would be that we could gain access to all the symbols of a sufficiently complete Unicode font. Apparently, current versions of Sphinx don't officially support either of these engines yet. And things don't work out of the box. But perhaps things can be made to work with a bit of patching. If so, that might greatly improve the situation.

See also #14153

CC: @nexttime @embray @jdemeyer @hivert

Component: documentation

Keywords: unicode

Issue created by migration from https://trac.sagemath.org/ticket/18370

@gagern gagern mannequin added this to the sage-6.7 milestone May 5, 2015
@gagern gagern mannequin added c: documentation labels May 5, 2015
@fchapoton

This comment has been minimized.

@fchapoton
Copy link
Contributor

Changed keywords from none to unicode

@fchapoton
Copy link
Contributor

comment:3

xelatex can be used with sphinx >= 1.5 (#22252)

see sphinx-doc/sphinx#894

@fchapoton
Copy link
Contributor

comment:4

any progress here ?

@dimpase
Copy link
Member

dimpase commented Aug 25, 2023

It's also needed in the developer manual. More and more tools output Unicode nowadays, including emoji chars, so one has to manually remove Unicode, as had to be done on #36117, for instance.

@dimpase dimpase changed the title Unicode support in reference manual Unicode support in reference manual and other docs, e.g. developer manual Aug 25, 2023
vbraun pushed a commit to vbraun/sage that referenced this issue Dec 23, 2023
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

to support sagemath#36857.

There have been numerous problems due to limited support of unicode in
pdflatex. I see no reason why we should not switch to modern tex engine
lualatex, as suggested in

https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-
latex_engine

We set lualatex as the default engine for
- building the sage documentation, in `sage_docbuild/`
- rendering pdf images, in `src/sage/misc/latex.py`

In  `src/sage/misc/latex.py`, the default engine is determined by
availability of the engines  with the order of preference: lualatex,
xelatex, pdflatex.

Along the way, we
- remove latex-related code deprecated in sagemath#32650.
- make lots of cosmetic edits in `src/sage/misc/latex.py`
- add support "lualatex" as an alternative to "[pdf|xe]latex" in
`src/sage/misc/latex.py`

We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy`
as new dependencies of `sagemath_doc_pdf` in addition to `texlive`
optional package. It seems that depending on the version and the
platform, `texlive` may already include the new dependencies. If not,
you need to install the new dependencies (as `_recommended` by the dummy
package)

On Ubuntu, the requirement is fulfilled by
```
sudo apt install texlive texlive-luatex fonts-freefont-otf xindy
```

The built pdf doc is available here: https://deploy-livedoc--
sagemath.netlify.app

To test `view` via lualatex, for example,
```
latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof
ont{Arial}")
view('Εύρηκα')
```
you should build sage on your local platform with this PR. You may need
to install `texlive-full` before testing. It seems that `texlive` (2019)
is not enough even for the `view` via pdftex.

sage-devel thread seeking Unicode testers:
https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I


<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
Resolves sagemath#18370
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36861
Reported by: Kwankyu Lee
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this issue Dec 24, 2023
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

to support sagemath#36857.

There have been numerous problems due to limited support of unicode in
pdflatex. I see no reason why we should not switch to modern tex engine
lualatex, as suggested in

https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-
latex_engine

We set lualatex as the default engine for
- building the sage documentation, in `sage_docbuild/`
- rendering pdf images, in `src/sage/misc/latex.py`

In  `src/sage/misc/latex.py`, the default engine is determined by
availability of the engines  with the order of preference: lualatex,
xelatex, pdflatex.

Along the way, we
- remove latex-related code deprecated in sagemath#32650.
- make lots of cosmetic edits in `src/sage/misc/latex.py`
- add support "lualatex" as an alternative to "[pdf|xe]latex" in
`src/sage/misc/latex.py`

We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy`
as new dependencies of `sagemath_doc_pdf` in addition to `texlive`
optional package. It seems that depending on the version and the
platform, `texlive` may already include the new dependencies. If not,
you need to install the new dependencies (as `_recommended` by the dummy
package)

On Ubuntu, the requirement is fulfilled by
```
sudo apt install texlive texlive-luatex fonts-freefont-otf xindy
```

The built pdf doc is available here: https://deploy-livedoc--
sagemath.netlify.app

To test `view` via lualatex, for example,
```
latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof
ont{Arial}")
view('Εύρηκα')
```
you should build sage on your local platform with this PR. You may need
to install `texlive-full` before testing. It seems that `texlive` (2019)
is not enough even for the `view` via pdftex.

sage-devel thread seeking Unicode testers:
https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I


<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
Resolves sagemath#18370
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36861
Reported by: Kwankyu Lee
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
vbraun pushed a commit to vbraun/sage that referenced this issue Dec 26, 2023
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

to support sagemath#36857.

There have been numerous problems due to limited support of unicode in
pdflatex. I see no reason why we should not switch to modern tex engine
lualatex, as suggested in

https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-
latex_engine

We set lualatex as the default engine for
- building the sage documentation, in `sage_docbuild/`
- rendering pdf images, in `src/sage/misc/latex.py`

In  `src/sage/misc/latex.py`, the default engine is determined by
availability of the engines  with the order of preference: lualatex,
xelatex, pdflatex.

Along the way, we
- remove latex-related code deprecated in sagemath#32650.
- make lots of cosmetic edits in `src/sage/misc/latex.py`
- add support "lualatex" as an alternative to "[pdf|xe]latex" in
`src/sage/misc/latex.py`

We add optional (dummy) packages `texlive_luatex`, `free_fonts`, `xindy`
as new dependencies of `sagemath_doc_pdf` in addition to `texlive`
optional package. It seems that depending on the version and the
platform, `texlive` may already include the new dependencies. If not,
you need to install the new dependencies (as `_recommended` by the dummy
package)

On Ubuntu, the requirement is fulfilled by
```
sudo apt install texlive texlive-luatex fonts-freefont-otf xindy
```

The built pdf doc is available here: https://deploy-livedoc--
sagemath.netlify.app

To test `view` via lualatex, for example,
```
latex.extra_preamble(r"\usepackage{fontspec}\setmainfont{Arial}\setmonof
ont{Arial}")
view('Εύρηκα')
```
you should build sage on your local platform with this PR. You may need
to install `texlive-full` before testing. It seems that `texlive` (2019)
is not enough even for the `view` via pdftex.

sage-devel thread seeking Unicode testers:
https://groups.google.com/g/sage-devel/c/tG2LK6Jvw0I


<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
Resolves sagemath#18370
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36861
Reported by: Kwankyu Lee
Reviewer(s): Dima Pasechnik, Kwankyu Lee, Matthias Köppe
@mkoeppe mkoeppe added this to the sage-10.3 milestone Dec 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants