Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build and deploy tutorials in CI #1487

Closed
fweik opened this issue Oct 5, 2017 · 5 comments · Fixed by #3024
Closed

Build and deploy tutorials in CI #1487

fweik opened this issue Oct 5, 2017 · 5 comments · Fixed by #3024
Assignees
Labels
Milestone

Comments

@fweik
Copy link
Contributor

fweik commented Oct 5, 2017

The job on gitlab ci that builds and deploys sphinx and doxygen should also do the same with the tutorial pdf's.

@fweik fweik added the DevOps label Oct 5, 2017
@RudolfWeeber RudolfWeeber added this to the Espresso 4.1 milestone Oct 15, 2018
@fweik fweik added this to To do in Espresso 4.1 Apr 19, 2019
@jngrad
Copy link
Member

jngrad commented Jul 19, 2019

Deploying the pdf's should not be too difficult. However, since this issue was opened, most of the tutorials have been rewritten as Jupyter notebooks. These need to be run in pypresso, which takes quite some time and requires some form of pre-processing to remove long or infinite loops and calls to the visualizers. I've been working on a solution to that problem using a more elaborate version of an already available pre-processing script importlib_wrapper.py.

@jngrad
Copy link
Member

jngrad commented Jul 20, 2019

Once we get the pdf's deployed, we might as well remove them from the source code. They always get outdated.

@jngrad
Copy link
Member

jngrad commented Jul 22, 2019

Update: to build PDF's, we need texlive installed. I tried texlive (240 MB), texlive-latex-extra (404 MB) and texlive-full (4.7 GB) and only the latter works. When using the formers, some essential packages are missing (e.g. TikZ) and cannot be installed via the texlive package manager due to a difference of year; I tried pinning the 2017 version of the texlive archive maintained on TUG, only to get error messages (see trace below). Installing them with wget is too time-consuming.
@mkuron could an extra 4.7 GB of data in the cuda-tutorials image cause performance issue?

$ tlmgr install tikz
(running on Debian, switching to user mode!)
tlmgr: Remote repository is newer than local (2017 < 2019)
Cross release updates are only supported with
  update-tlmgr-latest(.sh/.exe) --update
Please see https://tug.org/texlive/upgrade.html for details.
$ tlmgr option repository ftp://tug.org/historic/systems/texlive/2017/tlnet-final
(running on Debian, switching to user mode!)
tlmgr: setting default package repository to ftp://tug.org/historic/systems/texlive/2017/tlnet-final
$ tlmgr install tikz
(running on Debian, switching to user mode!)
/usr/bin/tlmgr: open tlpdb(ftp://tug.org/historic/systems/texlive/2017/tlnet-final/tlpkg/texlive.tlpdb) failed:  at /usr/share/texlive/tlpkg/TeXLive/TLPDB.pm line 362.
$ tlmgr install revtex4-1
(running on Debian, switching to user mode!)
tlmgr: main database at ftp://tug.org/historic/systems/texlive/2017/tlnet-final is not signed

@mkuron
Copy link
Member

mkuron commented Jul 22, 2019

I wouldn’t recommend increasing the image size like that. How about installing a minimal texlive 2019 with the needed packages? There is a ppa at https://launchpad.net/~jonathonf/+archive/ubuntu/texlive.

@jngrad
Copy link
Member

jngrad commented Jul 23, 2019

Cool! With this PPA, the image size only increases by 240 MB (177 MB for texlive-latex-recommended and the rest for required packages). This increases the coupling between the docker and espresso repos though (third block below), so we should refrain ourselves from creating new tutorials in LaTeX:

# install TeXlive 2019 from a PPA
RUN apt-get update \
    && apt-get install -y --no-install-recommends software-properties-common \
    && add-apt-repository -y ppa:jonathonf/texlive \
    && apt-get update \
    && apt-get install -y --no-install-recommends texlive-latex-recommended texlive-fonts-recommended

RUN useradd -m espresso
USER 1000

# install extra LaTeX packages locally
RUN tlmgr init-usertree \
    && tlmgr install pgf tikz-3dplot revtex units mhchem chemgreek todonotes upquote framed subfigure cleveref \
    && tlmgr install lm stmaryrd 2>&1 || : \
    && updmap -user

@jngrad jngrad self-assigned this Jul 24, 2019
bors bot added a commit that referenced this issue Jul 31, 2019
3024: Build and deploy tutorials in CI r=KaiSzuttor a=jngrad

Fixes #1487

Description of changes:
- introduce a CUDA 10.1 job
- simplify the export statements for notebooks with a single `nb_export` macro:
    - conversion to Python scripts is mandatory for tutorial tests
    - conversion to HTML (with or without running cells) is mandatory for the website
    - having the `html_export`/`python_export` macro pair caused code duplication in CMake
- CI/CD:
    - compile and deploy LaTeX tutorials as PDF to the website
    - run and deploy IPython tutorials as HTML to the website
    - don't deploy IPython tutorials as Python to the website
    - url: http://espressomd.org/wordpress/documentation/
- cleanup of LaTeX tutorials:
    - remove regenerable files (i.e. .pdf and .toc files) since they were always outdated and are now generated by CI/CD
    - remove the LaTeX template subfolder, since we decidedto write future tutorials in Jupyter
- the steps required to create a new Jupyter tutorial are now [explained in the wiki](https://github.com/espressomd/espresso/wiki/Documentation)

Co-authored-by: Jean-Noël Grad <jgrad@icp.uni-stuttgart.de>
@bors bors bot closed this as completed in #3024 Jul 31, 2019
Espresso 4.1 automation moved this from To do to Done Jul 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Espresso 4.1
  
Done
Development

Successfully merging a pull request may close this issue.

4 participants