Upload error to Rpubs from R markdown Error in rawToChar(response$headers) : argument 'x' must be a raw vector

[billy-s-lab]

Hi,

I am trying to publish an RMarkdown file to RPubs using rsconnect.

The file knits to HTML appropriately on my own laptop. When I try to publish the knitted HTML to RPubs, I get the following error:

Error in rawToChar(response$headers) : argument 'x' must be a raw vector

Screenshot of error below:

I have been able to publish other RMarkdown documents to RPubs in the past few days.

Any help greatly appreciated. Thank you.

sessionInfo() below:

R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS High Sierra 10.13.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] RColorBrewer_1.1-2 pheatmap_1.0.12 factoextra_1.0.7 FactoMineR_2.3 gridExtra_2.3
[6] mgcv_1.8-31 nlme_3.1-147 gpairs_1.3.3 scales_1.1.1 kableExtra_1.1.0
[11] knitr_1.28 cmprsk_2.2-9 cowplot_1.0.0 survminer_0.4.6 survival_3.1-12
[16] lmerTest_3.1-2 wesanderson_0.3.6 PredictABEL_1.2-4 DescTools_0.99.35 lmtest_0.9-37
[21] zoo_1.8-8 lme4_1.1-23 Matrix_1.2-18 MASS_7.3-51.6 ggfortify_0.4.10
[26] GGally_1.5.0 ggrepel_0.8.2 ggsci_2.9 rstatix_0.6.0 ggpubr_0.3.0
[31] broom_0.5.6 forcats_0.5.0 stringr_1.4.0 dplyr_0.8.5 purrr_0.3.4
[36] readr_1.3.1 tidyr_1.1.0 tibble_3.0.1 ggplot2_3.3.0 tidyverse_1.3.0
[41] readxl_1.3.1 rsconnect_0.8.16

loaded via a namespace (and not attached):
[1] minqa_1.2.4 colorspace_1.4-1 ggsignif_0.6.0 ellipsis_0.3.1 rio_0.5.16
[6] htmlTable_1.13.3 base64enc_0.1-3 fs_1.4.1 rstudioapi_0.11 farver_2.0.3
[11] PBSmodelling_2.68.8 fansi_0.4.1 mvtnorm_1.1-0 lubridate_1.7.8 xml2_1.3.2
[16] leaps_3.1 splines_4.0.0 Formula_1.2-3 jsonlite_1.6.1 nloptr_1.2.2.1
[21] km.ci_0.5-2 cluster_2.1.0 dbplyr_1.4.3 png_0.1-7 compiler_4.0.0
[26] httr_1.4.1 backports_1.1.7 assertthat_0.2.1 cli_2.0.2 acepack_1.4.1
[31] htmltools_0.4.0 tools_4.0.0 gtable_0.3.0 glue_1.4.1 Rcpp_1.0.4.6
[36] carData_3.0-3 cellranger_1.1.0 vctrs_0.3.0 xfun_0.14 openxlsx_4.1.5
[41] rvest_0.3.5 lifecycle_0.2.0 statmod_1.4.34 XML_3.99-0.3 barcode_1.1
[46] hms_0.5.3 expm_0.999-4 curl_4.3 KMsurv_0.1-5 rpart_4.1-15
[51] reshape_0.8.8 latticeExtra_0.6-29 stringi_1.4.6 checkmate_2.0.0 boot_1.3-25
[56] zip_2.0.4 rlang_0.4.6 pkgconfig_2.0.3 evaluate_0.14 lattice_0.20-41
[61] ROCR_1.0-11 labeling_0.3 htmlwidgets_1.5.1 tidyselect_1.1.0 plyr_1.8.6
[66] magrittr_1.5 R6_2.4.1 generics_0.0.2 Hmisc_4.4-0 DBI_1.1.0
[71] pillar_1.4.4 haven_2.2.0 foreign_0.8-79 withr_2.2.0 scatterplot3d_0.3-41
[76] abind_1.4-5 nnet_7.3-14 modelr_0.1.8 crayon_1.3.4 car_3.0-7
[81] survMisc_0.5.5 rmarkdown_2.3 jpeg_0.1-8.1 grid_4.0.0 data.table_1.12.8
[86] flashClust_1.01-2 vcd_1.4-7 webshot_0.5.2 reprex_0.3.0 digest_0.6.25
[91] xtable_1.8-4 numDeriv_2016.8-1.1 munsell_0.5.0 viridisLite_0.3.0 tcltk_4.0.0
[96] fortunes_1.5-4

[jmcphers]

I have been able to publish other RMarkdown documents to RPubs

Since other documents are publishing without issue, we probably won't be able to figure out what's going on without looking at the actual doc you're trying to publish. Are you able to share that?

[bpbond]

I'm also experiencing this error, and would welcome any insight into what's causing it.

My RMarkdown file is here, and session information is below. Thanks!

R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] emo_0.0.0.9000     ggplot2_3.3.3      RColorBrewer_1.1-2

loaded via a namespace (and not attached):
 [1] tidyselect_1.1.0  xfun_0.19         remotes_2.2.0     purrr_0.3.4      
 [5] colorspace_2.0-0  vctrs_0.3.6       generics_0.1.0    testthat_3.0.1   
 [9] htmltools_0.5.0   usethis_2.0.0     viridisLite_0.3.0 yaml_2.2.1       
[13] utf8_1.1.4        rlang_0.4.10      pkgbuild_1.2.0    pillar_1.4.7     
[17] glue_1.4.2        withr_2.3.0       sessioninfo_1.1.1 lifecycle_0.2.0  
[21] stringr_1.4.0     munsell_0.5.0     gtable_0.3.0      devtools_2.2.1   
[25] evaluate_0.14     memoise_1.1.0     labeling_0.4.2    knitr_1.30       
[29] callr_3.5.1       ps_1.5.0          curl_4.3          fansi_0.4.1      
[33] Rcpp_1.0.5        scales_1.1.1      desc_1.2.0        pkgload_1.1.0    
[37] farver_2.0.3      fs_1.5.0          digest_0.6.27     stringi_1.5.3    
[41] processx_3.4.5    dplyr_1.0.2       grid_3.6.1        rprojroot_2.0.2  
[45] cli_2.2.0         tools_3.6.1       magrittr_2.0.1    tibble_3.0.4     
[49] crayon_1.3.4      pkgconfig_2.0.3   ellipsis_0.3.1    rsconnect_0.8.16 
[53] prettyunits_1.1.1 lubridate_1.7.9.2 assertthat_0.2.1  rmarkdown_2.5    
[57] rstudioapi_0.13   R6_2.5.0          compiler_3.6.1  

[bpbond]

Another data point: my colleague @stephpenn1 is running R 4.03 and tried to publish this same Rmd file:

Error in curl::curl_fetch_memory(url, handle = handle) : 
  HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
Calls: <Anonymous> ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>

[aronatkins]

This is also reported as a curl issue here: jeroen/curl#232

I have been able to recreate this problem using the Rmd at https://github.com/JGCRI/Rworkshops/blob/master/ggplot2-basics.Rmd. The issue appears to be related to the size of the generated content - anything smaller than ~16M uploads successfully.

With that ggplot2-basics.Rmd, I can successfully deploy by removing everything below the ## 6. The ggplot call heading. Not a workaround or solution, but maybe a lead.

[aronatkins]

The code in question, where we set some large-file parameters and then send the rpubs request:

rsconnect/R/http-libcurl.R

Lines 81 to 128 in 8c11a55

	if (identical(method, "POST")) {
	# for POST operations, send all the file's content up at once. this is necessary because some
	# POST endpoints return 3xx status codes, which require a seekfunction in order to replay the
	# payload (the curl package does not currently allow specifying seekfunctions from R)
	curl::handle_setopt(handle,
	post = TRUE,
	postfields = readBin(con,
	what = "raw",
	n = fileLength),
	postfieldsize_large = fileLength)
	} else if (identical(method, "PUT")) {
	# for PUT operations, which are often used for larger content (bundle uploads), stream the
	# file from disk instead of reading it from memory
	curl::handle_setopt(handle,
	upload = TRUE,
	infilesize_large = fileLength)
	curl::handle_setopt(handle,
	readfunction = function(nbytes, ...) {
	if (is.null(con)) {
	return(raw())
	}
	bin <- readBin(con, "raw", nbytes)
	if (length(bin) < nbytes) {
	close(con)
	con <<- NULL
	}
	bin
	})
	} else {
	# why was a file specified for this endpoint?
	warning("Content file specified, but not used because the '", method, "' request ",
	"type does not accept a body.")
	}
	}
	# ensure we're using the requested method
	curl::handle_setopt(handle, customrequest = method)
	# apply all our accumulated headers
	curl::handle_setheaders(handle, .list = headers)
	# make the request
	response <- NULL
	time <- system.time(gcFirst = FALSE, tryCatch({
	# fetch the response into a raw buffer in memory
	response <- curl::curl_fetch_memory(url, handle = handle)
	},

[bpbond]

Ah, thanks @aronatkins ! Hopefully this is useful for you guys, but for my case, I should be able to reduce the size of the content and thus upload.

[aronatkins]

More information: If I disable HTTP/2 with the following code inserted into httpLibCurl:

  # disable http/2, per https://cran.r-project.org/web/packages/curl/vignettes/intro.html#disabling_http2
  curl::handle_setopt(handle, http_version = 2)

I get the original error reported in this issue:

Error in rawToChar(response$headers) : argument 'x' must be a raw vector

Running through in the debugger, this is because response is NULL. The response is NULL because we are falling into the block here:

rsconnect/R/http-libcurl.R

Lines 131 to 133 in 8c11a55

	if (identical(e$message, "Callback aborted") ||
	identical(e$message, "transfer closed with outstanding read data remaining"))
	return(NULL)

The e error object:

<simpleError in curl::curl_fetch_memory(url, handle = handle): transfer closed with outstanding read data remaining>

@jmcphers - It feels as if we have at least one error path in httpLibCurl that is not trapped, but underlying cause may be something about the rpubs server response.

[jeroen]

What httpd is rpubs running? transfer closed with outstanding read data remaining could mean that your server is setting an invalid content-length on the response headers. Which could happen for example when you are mixing up gzipped and non-gzipped responses in your cache.

[aronatkins]

I think I've found that the rpubs server is erring requests that send too-large uploads, and does so without reading the full request body. That feels OK, so we may simply need to react to those transfer errors in rsconnect (possibly preventing them when we know the upload is too large).

[hadley]

This Rmd is about ~17 Mb on my computer:

---
output: html_document
---

```{r}
library(ggplot2)

plot <- ggplot(diamonds, aes(carat, price)) + geom_point()
for (i in 1:80) {
  print(plot)
}
```

And I published it with:

rpubsUpload("large-file", "~/Desktop/ggplot2.html", "~/Desktop/ggplot2.Rmd")