Images also need pandoc #10

gaborcsardi · 2016-09-06T07:44:35Z

Pandoc is in the SystemRequirements of knitr:
https://github.com/cran/knitr/blob/e3adb6f1d7b1ed4ae9c016c015c88e5a237f1d3f/DESCRIPTION#L90-L93

We already have an entry for it in sysreqs DB:
https://github.com/r-hub/sysreqsdb/blob/master/sysreqs/pandoc.json

Maybe Suggests is not used for finding system requirements?

The text was updated successfully, but these errors were encountered:

jeroen · 2016-09-06T07:46:49Z

But from the log is looks pandoc, knitr and rmarkdown are all getting installed, and it still gives the warning.

@jimhester

gaborcsardi · 2016-09-06T07:48:36Z

@jeroenooms Oh, indeed.

jeroen · 2016-09-06T07:50:49Z

The build fails due to an xvfb problem by the way, not this warning.

jimhester · 2016-09-06T11:31:13Z

From https://github.com/yihui/knitr/blob/9394e12b764c9729368200a04dae770c098ab808/R/utils-vignettes.R#L8-L24 it needs pandoc-citeproc as well, which doesn't look to be installed on R-hub. https://builder.r-hub.io/status/magick_0.3.tar.gz-12769ec3c1339f46527af5bad4b3f1ca#L174 is the only mention in your build log.

gaborcsardi · 2016-09-06T12:48:55Z

I think knitr should explicitly specify that it needs pandoc-citeproc then. Anyway, I'll put in an exception for it in a minute....

gaborcsardi · 2016-09-06T12:51:55Z

Or do you think we should always install pandoc-citeproc, whenever we install pandoc? Maybe that's the best solution....

jimhester · 2016-09-06T13:10:44Z

@gaborcsardi I think you are probably correct, pandoc-citeproc should probably be in SystemRequirements: of knitr or rmarkdown (or both). These are really more like SuggestedSystemRequirements of course, but there isn't a defined field for optional system dependencies...

gaborcsardi · 2016-09-06T13:19:44Z

OK, I think we'll just install pandoc-citeproc (it is quite small) whenever pandoc is wanted.... but I need to update the sysreqs web app for that, because apparently it does not handle lists of packages....

jeroen · 2016-09-06T15:38:21Z

Crossposting here: yihui/knitr#1286

gaborcsardi · 2016-09-06T15:40:12Z

@jeroenooms wait, so the missing pandoc-citeproc is not the problem for us? I thought it was.

jeroen · 2016-09-06T15:42:00Z

I think in this case it is, but I am seeing this problem locally as well sometimes (and I have both...)

jeroen · 2016-09-06T15:42:53Z

Did you make pandoc-citeproc part of the default image? Then I'll try again if it now works...

gaborcsardi · 2016-09-06T15:44:39Z

Not the default image, but I'll just install pandoc-citeproc, whenever
pandoc is requested.

But this is not done yet, something in sysreqs does not handle package
lists, so I need to fix that first. I'll comment here, once it is done.

On Tue, Sep 6, 2016 at 4:42 PM, Jeroen Ooms notifications@github.com
wrote:

Did you make pandoc-citeproc part of the default image? Then I'll try
again if it now works...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAoTQLXMxZxtjdifRkLv3P8KskTEJIFUks5qnYn9gaJpZM4J1jQe
.

gaborcsardi · 2016-09-06T21:03:53Z

It should be OK now, pandoc and pandoc-citeproc go together.

gaborcsardi · 2016-09-06T21:04:39Z

E.g. https://builder.r-hub.io/status/magick_0.3.tar.gz-5a02bf692c10477b02fd904a98b480a3

gaborcsardi · 2016-09-06T21:06:06Z

Btw. these links are not very good on GitHub, because they are removed after 3 days, I mean r-hub removed the builds.... just FYI....

jeroen · 2016-09-12T12:42:52Z

Did you fix this yet? I'm still getting these warnings: https://builder.r-hub.io/status/curl_1.9000.tar.gz-9a07cdd4f9607723c5e9b8f135db8376

gaborcsardi · 2016-09-12T12:47:36Z

I did, but apparently EPEL does not have a pandoc-citeproc package. Or is it called differently? Then we need to update this entry: https://github.com/r-hub/sysreqsdb/blob/master/sysreqs/pandoc.json

jeroen · 2016-09-12T12:52:58Z

Hmm you're right, it's in EPEL 7 but not EPEL 6: https://apps.fedoraproject.org/packages/pandoc-citeproc

gaborcsardi · 2016-09-12T13:01:21Z

If it is available in some other repo, backported or sg., then I can add that repo. Or we need to compile it? Or download a binary from somewhere?

jeroen · 2016-09-12T13:14:46Z

Meh I think you'd have to install from source. Or we could host backports somewhere on a gh-pages branch or so....

jeroen · 2016-09-12T13:16:01Z

It's not clear to me why we need citeproc in the first place for vignettes without citations. @yihui

yihui · 2016-09-12T14:04:22Z

Just In case your vignettes have bibliography.

jimhester · 2016-09-12T15:31:26Z

Travis simply installs Pandoc from the .deb release, which includes pandoc-citeproc https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/r.rb#L473

jeroen · 2016-09-12T15:48:31Z

Yeah but centos 6 doesn't have that...

yihui · 2016-09-12T15:57:32Z

You can (pre)build both pandoc and pandoc-citeproc as standalone binaries.

gaborcsardi · 2016-09-12T17:38:03Z

I can build an rpm for it, and put it in a repo I suppose. No need to put these on GH pages, they will be fine on r-hub.

jeroen · 2016-10-13T16:58:23Z

@gaborcsardi did you end up creating a pandoc-citeproc rpm for CentOS6? I would like a copy of that :D

gaborcsardi · 2016-10-14T01:24:40Z

It is not just that. pandoc in centos6 is very old, that needs to be updated, too. So I am working out a way to do this. :/

Also, wch/r-source@74c8486

gaborcsardi · 2016-10-14T01:33:41Z

I'll install it from the RStudio binaries, and via a custom install specification in sysreqs (to be added).

So no RPM, sorry.....

gaborcsardi · 2016-10-24T22:49:20Z

OK, so I am downloading and installing pandoc from the RStudio-built binaries, and still have issues. :/

Fedora:

/usr/local/bin/pandoc: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory
Error in system(paste(shQuote(pandoc_path), "--version"), intern = TRUE) : 
  error in running command

Ubuntu:

/usr/local/bin/pandoc +RTS -K512m -RTS file8c0380f6b8b.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output file8c0380f6b8b.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /home/docker/R/rmarkdown/rmd/h/default.html --variable 'theme:bootstrap' --include-in-header /tmp/RtmplFNlOk/rmarkdown-str8c0440fe35a.html --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --no-highlight --variable highlightjs=/home/docker/R/rmarkdown/rmd/h/highlight --variable navigationjs=/home/docker/R/rmarkdown/rmd/h/navigation-1.1 
pandoc: Could not fetch https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css
HttpExceptionRequest Request {
  host                 = "maxcdn.bootstrapcdn.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/font-awesome/4.6.3/css/font-awesome.min.css"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getProtocolByName: does not exist (no such protocol name: tcp))
Error: pandoc document conversion failed with error 67
Execution halted

CentOS:

/usr/local/bin/pandoc +RTS -K512m -RTS filec3836351008.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output filec3836351008.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /home/docker/R/rmarkdown/rmd/h/default.html --variable 'theme:bootstrap' --include-in-header /tmp/Rtmpj3riqi/rmarkdown-strc38f5af397.html --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --no-highlight --variable highlightjs=/home/docker/R/rmarkdown/rmd/h/highlight --variable navigationjs=/home/docker/R/rmarkdown/rmd/h/navigation-1.1 
pandoc: Could not fetch https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/../fonts/fontawesome-webfont.eot?v=4.6.3
StatusCodeException (Status {statusCode = 403, statusMessage = "Forbidden"}) [("Server","nginx"),("Date","Mon, 24 Oct 2016 22:27:46 GMT"),("Content-Type","application/xml"),("Transfer-Encoding","chunked"),("Connection","keep-alive"),("Content-Encoding","gzip"),("X-Cache","MISS"),("X-Response-Body-Start","<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>57327A02ACC77868</RequestId><HostId>D59raUW8YitWFC2ovDF9FJupw95xOZcZyH4ETWQh4bN38P7XI/XTV+bICHKnQ77nlbgsyvMQ8qU=</HostId></Error>"),("X-Request-URL","GET https://maxcdn.bootstrapcdn.com:443/font-awesome/4.6.3/css/../fonts/fontawesome-webfont.eot?v=4.6.3")] (CJ {expose = []})
Error: pandoc document conversion failed with error 67
Execution halted

Debian is OK.

yihui · 2016-10-25T01:40:30Z

@gaborcsardi Can you provide the Rmd file and your rmarkdown version? AFAIK we don't use any https resources in rmarkdown, so I don't know why Pandoc was trying to download resources from bootstrapcdn.com.

gaborcsardi · 2016-10-25T01:53:56Z

@yihui It was the rchallenge package, as on CRAN. Yes, it is not you, it is an extension.

Btw. it needs libgmp and libffi on a stock Fedora 24 Docker container, it seems that the RPM versions assume old libs.

[root@32c294495d65 ~]# ldd /usr/local/bin/pandoc
    linux-vdso.so.1 (0x00007ffca0fde000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fa37ec43000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fa37ea3b000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fa37e837000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fa37e633000)
    libgmp.so.3 => not found
    libffi.so.5 => not found
    libm.so.6 => /lib64/libm.so.6 (0x00007fa37e328000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fa37df65000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa37dd48000)
    /lib64/ld-linux-x86-64.so.2 (0x000055d5b7335000)

yihui · 2016-10-25T02:08:39Z

Okay, I'd recommend rchallenge either to localize font-awesome, or not to use the self-contained mode. It is a waste of time and resources to download a copy of font-awesome every time a document is compiled.

I don't know how RStudio's pandoc was built, but I wonder if you have tried to build the binary by yourself. You only need to build it once, and the self-contained binary can be stored elsewhere (S3, Github releases, ...) and used later. I guess it is not that terrible.

gaborcsardi · 2016-10-30T22:55:50Z

@jeroenooms pandoc RPMs for centos6: https://files.r-hub.io/pandoc/centos6/
Very minimal and much sweet.

gaborcsardi · 2016-10-30T23:00:01Z

@jeroenooms And if you want to see how they are built, or tweak: https://github.com/r-hub/pandoc-builders

gaborcsardi · 2016-10-30T23:06:22Z

@yihui

I don't know how RStudio's pandoc was built, but I wonder if you have tried to build the binary by yourself. You only need to build it once, and the self-contained binary can be stored elsewhere (S3, Github releases, ...) and used later. I guess it is not that terrible.

Yes, this would not be terrible, but unfortunately it is not true. It is hard to build a static binary, and the shared binary will depend on the system libs (libgmp in particular), so it won't be portable. E.g. RStudio's latest RPM binary does not work on Fedora 24, and the CentOS6 binary I built does not, either.

So in practice, I'll need to build at least a couple of binaries. Anyway, I have an "easy" way of building them now, at least.

gaborcsardi · 2016-10-31T12:02:53Z

@jeroenooms Updated (reverted) sysreqs, and updated centos6 images to install from our own repository:

rhub-linux-builders/centos6-epel/Dockerfile

Lines 9 to 11 in d09595f

    
           RUN \ 
        
             curl -o /etc/yum.repos.d/rhub-pandoc-centos6.repo                   \ 
        
               https://files.r-hub.io/pandoc/centos6/rhub-pandoc-centos6.repo

It all works fine now. And you get the latest pandoc and pandoc-citeproc, with HTTPS support.

gaborcsardi closed this as completed Sep 6, 2016

gaborcsardi reopened this Oct 24, 2016

gaborcsardi mentioned this issue Oct 25, 2016

Build proper pandoc packages #15

Closed

gaborcsardi closed this as completed Oct 25, 2016

Images also need pandoc #10

Images also need pandoc #10

Comments

gaborcsardi commented Sep 6, 2016 • edited Loading

jeroen commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

jeroen commented Sep 6, 2016 • edited Loading

jimhester commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

jimhester commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

jeroen commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

jeroen commented Sep 6, 2016

jeroen commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016

gaborcsardi commented Sep 6, 2016 • edited Loading

jeroen commented Sep 12, 2016

gaborcsardi commented Sep 12, 2016 • edited Loading

jeroen commented Sep 12, 2016 • edited Loading

gaborcsardi commented Sep 12, 2016

jeroen commented Sep 12, 2016

jeroen commented Sep 12, 2016

yihui commented Sep 12, 2016

jimhester commented Sep 12, 2016

jeroen commented Sep 12, 2016

yihui commented Sep 12, 2016

gaborcsardi commented Sep 12, 2016

jeroen commented Oct 13, 2016

gaborcsardi commented Oct 14, 2016

gaborcsardi commented Oct 14, 2016

gaborcsardi commented Oct 24, 2016

yihui commented Oct 25, 2016

gaborcsardi commented Oct 25, 2016

yihui commented Oct 25, 2016

gaborcsardi commented Oct 30, 2016

gaborcsardi commented Oct 30, 2016

gaborcsardi commented Oct 30, 2016 • edited Loading

gaborcsardi commented Oct 31, 2016

gaborcsardi commented Sep 6, 2016 •

edited

Loading

jeroen commented Sep 6, 2016 •

edited

Loading

gaborcsardi commented Sep 6, 2016 •

edited

Loading

gaborcsardi commented Sep 12, 2016 •

edited

Loading

jeroen commented Sep 12, 2016 •

edited

Loading

gaborcsardi commented Oct 30, 2016 •

edited

Loading