Make PDF great again

[SrNetoChan]

Goal:

Facilitate the creation of PDFs for the main documents:
User manual
PyQGIS Cookbook
Training manual

Ticket(s): #

• Backport to LTR documentation is required (Forward porting to master actually)

[DelazJ]

Hi @SrNetoChan
Thanks to all who participate to the fix. I gave it a quick run and it works nicely for English build.
I tried to build French and:

1. I had to interact with the build, since I got some prompt in the process (we'd need to find how to skip/fix those prompts - I don't know whether they are due to wrong translations in French (we have hundreds of warnings when building html but I'm not sure it's related and we can't expect 100% clean translation).
2. at the end of the process, it failed. Checking the generated folders, I have the 3 pdf files (they look nice afaict and are complete) in the latex/fr folder but no 'pdf/fr' folder was created. Looks like the mkdir did not make through.

Another "concern" is the exclusion of processing_algs resulting in warnings but also missing urls in gui and expression chapters. These two chapters have a lot of connection with the algs.

[SrNetoChan]

Hi @SrNetoChan
Thanks to all who participate to the fix. I gave it a quick run and it works nicely for English build.
I tried to build French and:

1. I had to interact with the build, since I got some prompt in the process (we'd need to find how to skip/fix those prompts - I don't know whether they are due to wrong translations in French (we have hundreds of warnings when building html but I'm not sure it's related and we can't expect 100% clean translation).

It probably has to be with strange characters. @havatv had a bunch of callbacks in here:

https://github.com/qgis/QGIS-Documentation/blob/release_3.10/conf.py#L237

And I needed to add the Pi character. Not sure if we are doing it the right way, it feels strange to me that we need to do this, maybe I should try tex2pdf too, to see wish works best.

Can you share those prompt error messages?

2. at the end of the process, it failed. Checking the generated folders, I have the 3 pdf files (they look nice afaict and are complete) in the latex/fr folder but no 'pdf/fr' folder was created. Looks like the mkdir did not make through.

That is strange. Maybe it has something to do with the first errors. I confess I haven't tried with any other language. How did you run the command? make pdf?

Another "concern" is the exclusion of processing_algs resulting in warnings but also missing urls in gui and expression chapters. These two chapters have a lot of connection with the algs.

I started by removing the processing algorithm because of some issues in the taudem algorithms, but then it was also a question o space. With the algorithms the PDF has more 400 pages, going up to 1100 in total. The 2.18 PDFs didn't have the algorithms.

Bear in mind that, in the PDFs, links to other documents (like the training manual) will also be broken.

I spent a morning trying to create a conditional TOC that would move the processing algorithms to another document just for latex, keeping the user manual small, but having the algorithms available as an extra. Afterall, they are useful both for the user manual as for the python cookbook. But I have failed miserably. the only:: directive does not work well with the TOC, unless we use some customized directive, and that is a path that I really don't want to take.

We could move the processing algorithms to another document even in HTML, or simply include it all in the user's manual PDF. Whatever you guys prefer.

[SrNetoChan]

I tried creating the pt_PT PDFs and I got a bunch of errors. I had to install texlive-lang-all and it worked. I can't add it to pip requirements though...

[rduivenvoorde]

@SrNetoChan just try it out. That is what I do.

Note though that to make this work you will need a lot more to do:

• you need a machine (or a docker container) with all the locale's you need to build all the different TEX language charachters. qgis2 itself (or current docker) is too old for this:
  https://github.com/qgis/QGIS-Sysadmin/tree/master/docker/sphinx (you see a separate docker for html and one for pdf)
• earlier (but you have to dive into this deeper) we needed 2 or three different tex->pdf builders because one did not work for Korean, other not for Russian etc etc. That is the reason I did not open this can of worms yet :-)
  https://github.com/qgis/QGIS-Sysadmin/blob/master/docker/sphinx/Dockerfile-pdf

But feel free to just merge, but please let 'make site' not (yet) build pdf. If we have everything automated I'm ok to do that, but now it is easier (for me) to just run make site on one of my little home servers.

[SrNetoChan]

@rduivenvoorde @havatv @DelazJ my idea was just to allow people to create their own PDFs. I don't want to go back to build the PDFs for them.

Should we include the processing algorithms or not? I don't think people will print it, so having a 1100 pages PDF is not a problem to me.

Wouldn't be bad if we detached the algorithms as a different doc though.

[havatv]

@rduivenvoorde @havatv @DelazJ my idea was just to allow people to create their own PDFs. I don't want to go back to build the PDFs for them.

Making (reasonably) up-to-date PDFs available on docs.qgis.org would be a great service to users. I have been a user of these PDFs for years.

Should we include the processing algorithms or not? I don't think people will print it, so having a 1100 pages PDF is not a problem to me.

I agree about printing - a large number of pages is not a problem. And having the processing algs in the user manual PDF is convenient for searching.

Wouldn't be bad if we detached the algorithms as a different doc though.

This is also a possibility, but I would prefer having them in the user manual.

And thank you for digging into this!

[SrNetoChan]

@havatv in that case I will remove the algorithms exclusion and merge this PR.

After that, I will see how building PDFs in other languages work.

[SrNetoChan]

I have added make PDF to Travis, to ensure that the PDF build is not compromised by the introduction of some strange character. But then I realized that this will make travis checks to take much longer. Should I remove it?

[DelazJ]

But then I realized that this will make travis checks to take much longer. Should I remove it?

For travis, I think html generation for review is more important than pdf.
But let's see how long it takes and if it's too much, remove it.

Making (reasonably) up-to-date PDFs available on docs.qgis.org would be a great service to users.

Fully agree. I'm not sure that there are so many people other than contributors that have ever built the docs. Now that we have a how-to, we should try to provide at least English pdf (until we find a clean way to generate all the others).
I've looked at texlive-lang-all but it's too big, and texlive-lang-fr is not enough to fix the issues with French.
The docker route is probably worth it (i'm not suggesting it to you, Alex).

[SrNetoChan]

I removed the PDF tests. I don't know enough of Travis for it.

[havatv]

Great work!
Building PDF works like a charm on Ubuntu 18.04 without any preparations using

make LANG=en pdf

I also tested

make LANG=fr pdf

Got a lot of error messages, and the "compile" stopped with

(/usr/share/texlive/texmf-dist/tex/generic/babel/switch.def)

! Package babel Error: You haven't specified a language option.

And after doing q some times I ended up with

Makefile:57: recipe for target 'pdf' failed
make: *** [pdf] Error 1

Installing texlive-lang-french helped! :-)
The French language PDFs end up in build/latex/fr/. build/pdf/fr is not created.

One error that could be causing problems:

Package inputenc Error: Unicode char fi (U+FB01)

Adding

\\newunicodechar{fi}{fi}

to preamble in latex_element of conf.py did the trick, but I am not sure if it is the correct medicine.

The problem comes from the following po file:

locale/fr/LC_MESSAGES/docs/training_manual/basic_map/symbology.po:77:msgstr "|basic| |FA| Modification des couleurs"

(translated by https://www.transifex.com/user/profile/rduivenvoorde/ in March 2020).

Should be:

locale/fr/LC_MESSAGES/docs/training_manual/basic_map/symbology.po:77:msgstr "|basic| |FA| Modification des couleurs"

I became a member of the French transiflex team today, and changed "fi" to "fi". So the suggested conf.py update should no longer be necessary.