# Images also need pandoc #10

Closed
opened this Issue Sep 6, 2016 · 38 comments

Projects
None yet
4 participants
Member

### gaborcsardi commented Sep 6, 2016 • edited Edited 1 time gaborcsardi edited Sep 6, 2016 (most recent)

 Pandoc is in the SystemRequirements of knitr: https://github.com/cran/knitr/blob/e3adb6f1d7b1ed4ae9c016c015c88e5a237f1d3f/DESCRIPTION#L90-L93 We already have an entry for it in sysreqs DB: https://github.com/r-hub/sysreqsdb/blob/master/sysreqs/pandoc.json Maybe Suggests is not used for finding system requirements?
Member

### jeroen commented Sep 6, 2016

 But from the log is looks pandoc, knitr and rmarkdown are all getting installed, and it still gives the warning. @jimhester
Member

### gaborcsardi commented Sep 6, 2016

 @jeroenooms Oh, indeed.
Member

### jeroen commented Sep 6, 2016 • edited Edited 1 time jeroen edited Sep 6, 2016 (most recent)

 The build fails due to an xvfb problem by the way, not this warning.

### jimhester commented Sep 6, 2016

 From https://github.com/yihui/knitr/blob/9394e12b764c9729368200a04dae770c098ab808/R/utils-vignettes.R#L8-L24 it needs pandoc-citeproc as well, which doesn't look to be installed on R-hub. https://builder.r-hub.io/status/magick_0.3.tar.gz-12769ec3c1339f46527af5bad4b3f1ca#L174 is the only mention in your build log.
Member

### gaborcsardi commented Sep 6, 2016

 I think knitr should explicitly specify that it needs pandoc-citeproc then. Anyway, I'll put in an exception for it in a minute....
Member

### gaborcsardi commented Sep 6, 2016

 Or do you think we should always install pandoc-citeproc, whenever we install pandoc? Maybe that's the best solution....

### jimhester commented Sep 6, 2016

 @gaborcsardi I think you are probably correct, pandoc-citeproc should probably be in SystemRequirements: of knitr or rmarkdown (or both). These are really more like SuggestedSystemRequirements of course, but there isn't a defined field for optional system dependencies...
Member

### gaborcsardi commented Sep 6, 2016

 OK, I think we'll just install pandoc-citeproc (it is quite small) whenever pandoc is wanted.... but I need to update the sysreqs web app for that, because apparently it does not handle lists of packages....
Member

### jeroen commented Sep 6, 2016

 Crossposting here: yihui/knitr#1286
Member

### gaborcsardi commented Sep 6, 2016

 @jeroenooms wait, so the missing pandoc-citeproc is not the problem for us? I thought it was.
Member

### jeroen commented Sep 6, 2016

 I think in this case it is, but I am seeing this problem locally as well sometimes (and I have both...)
Member

### jeroen commented Sep 6, 2016

 Did you make pandoc-citeproc part of the default image? Then I'll try again if it now works...
Member

### gaborcsardi commented Sep 6, 2016

 Not the default image, but I'll just install pandoc-citeproc, whenever pandoc is requested. But this is not done yet, something in sysreqs does not handle package lists, so I need to fix that first. I'll comment here, once it is done. On Tue, Sep 6, 2016 at 4:42 PM, Jeroen Ooms notifications@github.com wrote: Did you make pandoc-citeproc part of the default image? Then I'll try again if it now works... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub #10 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AAoTQLXMxZxtjdifRkLv3P8KskTEJIFUks5qnYn9gaJpZM4J1jQe .
Member

### gaborcsardi commented Sep 6, 2016

 It should be OK now, pandoc and pandoc-citeproc go together.

Member

Member

### gaborcsardi commented Sep 6, 2016 • edited Edited 1 time gaborcsardi edited Sep 6, 2016 (most recent)

 Btw. these links are not very good on GitHub, because they are removed after 3 days, I mean r-hub removed the builds.... just FYI....
Member

### jeroen commented Sep 12, 2016

 Did you fix this yet? I'm still getting these warnings: https://builder.r-hub.io/status/curl_1.9000.tar.gz-9a07cdd4f9607723c5e9b8f135db8376
Member

### gaborcsardi commented Sep 12, 2016 • edited Edited 1 time gaborcsardi edited Sep 12, 2016 (most recent)

 I did, but apparently EPEL does not have a pandoc-citeproc package. Or is it called differently? Then we need to update this entry: https://github.com/r-hub/sysreqsdb/blob/master/sysreqs/pandoc.json
Member

### jeroen commented Sep 12, 2016 • edited Edited 1 time jeroen edited Sep 12, 2016 (most recent)

 Hmm you're right, it's in EPEL 7 but not EPEL 6: https://apps.fedoraproject.org/packages/pandoc-citeproc
Member

### gaborcsardi commented Sep 12, 2016

 If it is available in some other repo, backported or sg., then I can add that repo. Or we need to compile it? Or download a binary from somewhere?
Member

### jeroen commented Sep 12, 2016

 Meh I think you'd have to install from source. Or we could host backports somewhere on a gh-pages branch or so....
Member

### jeroen commented Sep 12, 2016

 It's not clear to me why we need citeproc in the first place for vignettes without citations. @yihui

### yihui commented Sep 12, 2016

 Just In case your vignettes have bibliography.

### jimhester commented Sep 12, 2016

 Travis simply installs Pandoc from the .deb release, which includes pandoc-citeproc https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/r.rb#L473
Member

### jeroen commented Sep 12, 2016

 Yeah but centos 6 doesn't have that...

### yihui commented Sep 12, 2016

 You can (pre)build both pandoc and pandoc-citeproc as standalone binaries.
Member

### gaborcsardi commented Sep 12, 2016

 I can build an rpm for it, and put it in a repo I suppose. No need to put these on GH pages, they will be fine on r-hub.
Member

### jeroen commented Oct 13, 2016

 @gaborcsardi did you end up creating a pandoc-citeproc rpm for CentOS6? I would like a copy of that :D
Member

### gaborcsardi commented Oct 14, 2016

 It is not just that. pandoc in centos6 is very old, that needs to be updated, too. So I am working out a way to do this. :/ Also, wch/r-source@74c8486
Member

### gaborcsardi commented Oct 14, 2016

 I'll install it from the RStudio binaries, and via a custom install specification in sysreqs (to be added). So no RPM, sorry.....

Member

### gaborcsardi commented Oct 24, 2016

 OK, so I am downloading and installing pandoc from the RStudio-built binaries, and still have issues. :/ Fedora: /usr/local/bin/pandoc: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory Error in system(paste(shQuote(pandoc_path), "--version"), intern = TRUE) : error in running command  Ubuntu: /usr/local/bin/pandoc +RTS -K512m -RTS file8c0380f6b8b.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output file8c0380f6b8b.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /home/docker/R/rmarkdown/rmd/h/default.html --variable 'theme:bootstrap' --include-in-header /tmp/RtmplFNlOk/rmarkdown-str8c0440fe35a.html --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --no-highlight --variable highlightjs=/home/docker/R/rmarkdown/rmd/h/highlight --variable navigationjs=/home/docker/R/rmarkdown/rmd/h/navigation-1.1 pandoc: Could not fetch https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css HttpExceptionRequest Request { host = "maxcdn.bootstrapcdn.com" port = 443 secure = True requestHeaders = [] path = "/font-awesome/4.6.3/css/font-awesome.min.css" queryString = "" method = "GET" proxy = Nothing rawBody = False redirectCount = 10 responseTimeout = ResponseTimeoutDefault requestVersion = HTTP/1.1 } (ConnectionFailure getProtocolByName: does not exist (no such protocol name: tcp)) Error: pandoc document conversion failed with error 67 Execution halted  CentOS: /usr/local/bin/pandoc +RTS -K512m -RTS filec3836351008.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output filec3836351008.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /home/docker/R/rmarkdown/rmd/h/default.html --variable 'theme:bootstrap' --include-in-header /tmp/Rtmpj3riqi/rmarkdown-strc38f5af397.html --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --no-highlight --variable highlightjs=/home/docker/R/rmarkdown/rmd/h/highlight --variable navigationjs=/home/docker/R/rmarkdown/rmd/h/navigation-1.1 pandoc: Could not fetch https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/../fonts/fontawesome-webfont.eot?v=4.6.3 StatusCodeException (Status {statusCode = 403, statusMessage = "Forbidden"}) [("Server","nginx"),("Date","Mon, 24 Oct 2016 22:27:46 GMT"),("Content-Type","application/xml"),("Transfer-Encoding","chunked"),("Connection","keep-alive"),("Content-Encoding","gzip"),("X-Cache","MISS"),("X-Response-Body-Start","\nAccessDeniedAccess Denied57327A02ACC77868D59raUW8YitWFC2ovDF9FJupw95xOZcZyH4ETWQh4bN38P7XI/XTV+bICHKnQ77nlbgsyvMQ8qU="),("X-Request-URL","GET https://maxcdn.bootstrapcdn.com:443/font-awesome/4.6.3/css/../fonts/fontawesome-webfont.eot?v=4.6.3")] (CJ {expose = []}) Error: pandoc document conversion failed with error 67 Execution halted  Debian is OK.

### yihui commented Oct 25, 2016

 @gaborcsardi Can you provide the Rmd file and your rmarkdown version? AFAIK we don't use any https resources in rmarkdown, so I don't know why Pandoc was trying to download resources from bootstrapcdn.com.
Member

### gaborcsardi commented Oct 25, 2016

 @yihui It was the rchallenge package, as on CRAN. Yes, it is not you, it is an extension. Btw. it needs libgmp and libffi on a stock Fedora 24 Docker container, it seems that the RPM versions assume old libs. [root@32c294495d65 ~]# ldd /usr/local/bin/pandoc linux-vdso.so.1 (0x00007ffca0fde000) libz.so.1 => /lib64/libz.so.1 (0x00007fa37ec43000) librt.so.1 => /lib64/librt.so.1 (0x00007fa37ea3b000) libutil.so.1 => /lib64/libutil.so.1 (0x00007fa37e837000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fa37e633000) libgmp.so.3 => not found libffi.so.5 => not found libm.so.6 => /lib64/libm.so.6 (0x00007fa37e328000) libc.so.6 => /lib64/libc.so.6 (0x00007fa37df65000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa37dd48000) /lib64/ld-linux-x86-64.so.2 (0x000055d5b7335000) 

### yihui commented Oct 25, 2016

 Okay, I'd recommend rchallenge either to localize font-awesome, or not to use the self-contained mode. It is a waste of time and resources to download a copy of font-awesome every time a document is compiled. I don't know how RStudio's pandoc was built, but I wonder if you have tried to build the binary by yourself. You only need to build it once, and the self-contained binary can be stored elsewhere (S3, Github releases, ...) and used later. I guess it is not that terrible.

Closed

Member

### gaborcsardi commented Oct 30, 2016

 @jeroenooms pandoc RPMs for centos6: https://files.r-hub.io/pandoc/centos6/ Very minimal and much sweet.
Member

### gaborcsardi commented Oct 30, 2016

 @jeroenooms And if you want to see how they are built, or tweak: https://github.com/r-hub/pandoc-builders
Member

### gaborcsardi commented Oct 30, 2016 • edited Edited 1 time gaborcsardi edited Oct 30, 2016 (most recent)

 @yihui I don't know how RStudio's pandoc was built, but I wonder if you have tried to build the binary by yourself. You only need to build it once, and the self-contained binary can be stored elsewhere (S3, Github releases, ...) and used later. I guess it is not that terrible. Yes, this would not be terrible, but unfortunately it is not true. It is hard to build a static binary, and the shared binary will depend on the system libs (libgmp in particular), so it won't be portable. E.g. RStudio's latest RPM binary does not work on Fedora 24, and the CentOS6 binary I built does not, either. So in practice, I'll need to build at least a couple of binaries. Anyway, I have an "easy" way of building them now, at least.
Member

### gaborcsardi commented Oct 31, 2016

@jeroenooms Updated (reverted) sysreqs, and updated centos6 images to install from our own repository:

Lines 9 to 11 in d09595f

 RUN \ curl -o /etc/yum.repos.d/rhub-pandoc-centos6.repo \ https://files.r-hub.io/pandoc/centos6/rhub-pandoc-centos6.repo

It all works fine now. And you get the latest pandoc and pandoc-citeproc, with HTTPS support.