New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkgdown consumed a huge amount (10+GB) of RAM on build_site() #783

Open
Robinlovelace opened this Issue Aug 13, 2018 · 8 comments

Comments

Projects
None yet
3 participants
@Robinlovelace

Robinlovelace commented Aug 13, 2018

I'm not sure how or why this happened but I'm confident it's pkgdown because I tried it twice (both times it crashed my R session).

Here's the crash message:

<simpleError in process_initialize(self, private, command, args, stdin, stdout,     stderr, connections, poll_connection, env, cleanup, cleanup_tree,     wd, echo_cmd, supervise, windows_verbatim_args, windows_hide_window,     encoding, post_process): Cannot fork Unknown error -12 at unix/processx.c:333>
Warning message:
system call failed: Cannot allocate memory 
Error in process_initialize(self, private, command, args, stdin, stdout,  : 
  Cannot fork Unknown error -12 at unix/processx.c:333

Here's the dramatic memory use log on Ubuntu:

image

With:

packageVersion("pkgdown")
#> [1] '1.1.0'

Created on 2018-08-13 by the reprex package (v0.2.0).

@Robinlovelace

This comment has been minimized.

Show comment
Hide comment
@Robinlovelace

Robinlovelace Aug 13, 2018

Live update: now trying with

pkgdown::build_site(new = F)

In a terminal to see if that solves it - seems to be something to do with running in parallel. Update - crazy RAM use happened again - didn't solve it.

Robinlovelace commented Aug 13, 2018

Live update: now trying with

pkgdown::build_site(new = F)

In a terminal to see if that solves it - seems to be something to do with running in parallel. Update - crazy RAM use happened again - didn't solve it.

@Robinlovelace

This comment has been minimized.

Show comment
Hide comment
@Robinlovelace

Robinlovelace Aug 13, 2018

In the name of reproducibility, I think others will be able to reproduce this by cloning this repo and running build_site() from in there: https://github.com/geocompr/geocompkg

I suspect the culprit is https://github.com/geocompr/geocompkg/blob/master/vignettes/point-pattern-heatmap.Rmd which has some leaflet code.

Robinlovelace commented Aug 13, 2018

In the name of reproducibility, I think others will be able to reproduce this by cloning this repo and running build_site() from in there: https://github.com/geocompr/geocompkg

I suspect the culprit is https://github.com/geocompr/geocompkg/blob/master/vignettes/point-pattern-heatmap.Rmd which has some leaflet code.

@Robinlovelace

This comment has been minimized.

Show comment
Hide comment
@Robinlovelace

Robinlovelace Aug 13, 2018

Update on the last attempt run from a terminal with new_process = F - it was definitely the R session itself that was munching the session because when it stopped R with quit() this happened:

image

Robinlovelace commented Aug 13, 2018

Update on the last attempt run from a terminal with new_process = F - it was definitely the R session itself that was munching the session because when it stopped R with quit() this happened:

image

Robinlovelace added a commit to geocompr/geocompkg that referenced this issue Aug 13, 2018

@Robinlovelace

This comment has been minimized.

Show comment
Hide comment
@Robinlovelace

Robinlovelace Aug 13, 2018

Another update: thinking maybe running it on a bigger computer with a different OS might help I tried it on my work desktop and get an error - the error message should shed more light on it though, hope this helps:

image

Robinlovelace commented Aug 13, 2018

Another update: thinking maybe running it on a bigger computer with a different OS might help I tried it on my work desktop and get an error - the error message should shed more light on it though, hope this helps:

image

@Robinlovelace

This comment has been minimized.

Show comment
Hide comment
@Robinlovelace

Robinlovelace Aug 13, 2018

In any case, for users, all the vignettes can be built by running build_articles(), that seems to run fine.

Robinlovelace commented Aug 13, 2018

In any case, for users, all the vignettes can be built by running build_articles(), that seems to run fine.

@Nowosad

This comment has been minimized.

Show comment
Hide comment
@Nowosad

Nowosad Aug 21, 2018

I also got this problem today. Vignette works well when knitted, but there is a memory leak while using build_site().

Nowosad commented Aug 21, 2018

I also got this problem today. Vignette works well when knitted, but there is a memory leak while using build_site().

@jayhesselberth

This comment has been minimized.

Show comment
Hide comment
@jayhesselberth

jayhesselberth Oct 6, 2018

Collaborator

Could you please build a minimal package that recreates the issue? The geocompkg package has too many dependencies to debug this effectively.

Collaborator

jayhesselberth commented Oct 6, 2018

Could you please build a minimal package that recreates the issue? The geocompkg package has too many dependencies to debug this effectively.

@Robinlovelace

This comment has been minimized.

Show comment
Hide comment
@Robinlovelace

Robinlovelace Oct 6, 2018

I think it was the number of vignettes, not the dependencies of the package the website of which we were trying to build, that caused the issue. Agree hard to debug.

Robinlovelace commented Oct 6, 2018

I think it was the number of vignettes, not the dependencies of the package the website of which we were trying to build, that caused the issue. Agree hard to debug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment