[R] try_download is confused when download.file.method isn't default

[asfimport]

Hello there and thanks again for this beautiful package!

I am trying to install arrow on linux and I got a few problematic warnings during the install. My computer is behind a firewall so not all the connections coming from rstudio are allowed.

 

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-ubuntu18-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS
Matrix products: default
BLAS/LAPACK: /apps/intel/2019.1/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so
locale:
 [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 
 [4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 
 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C 
[10] LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] MKLthreads_0.1
loaded via a namespace (and not attached):
[1] compiler_3.6.1 tools_3.6.1

 

after running install.packages("arrow") I get

 

 
installing *source* package ?arrow? ...
** package ?arrow? successfully unpacked and MD5 sums checked
** using staged installation
*** Successfully retrieved C++ source
*** Proceeding without C++ dependencies
Warning message:
In unzip(tf1, exdir = src_dir) : error 1 in extracting from zip file
./configure: line 132: cd: libarrow/arrow-0.17.1/lib: No such file or directory
------------------------- NOTE ---------------------------
After installation, please run arrow::install_arrow()
for help installing required runtime libraries
---------------------------------------------------------

 

 

However, the installation ends normally.

 

 ** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (arrow)

 

So I go ahead and try to run arrow::install_arrow() and get a similar warning.

 

installing *source* package ?arrow? ...
** package ?arrow? successfully unpacked and MD5 sums checked
** using staged installation
*** Successfully retrieved C++ binaries for ubuntu-18.04
Warning messages:
1: In file(file, "rt") :
 URL 'https://raw.githubusercontent.com/ursa-labs/arrow-r-nightly/master/linux/distro-map.csv': status was 'Couldn't connect to server'
2: In unzip(bin_file, exdir = dst_dir) :
 error 1 in extracting from zip file
./configure: line 132: cd: libarrow/arrow-0.17.1/lib: No such file or directory
------------------------- NOTE ---------------------------
After installation, please run arrow::install_arrow()
for help installing required runtime libraries

And unfortunately I cannot read any parquet file.

Error in fetch(key) : lazy-load database '/mydata/R/x86_64-ubuntu18-linux-gnu-library/3.6/arrow/help/arrow.rdb' is corrupt

 

Could you please tell me how to fix this? Can I just copy the zip from github and do a manual install in Rstudio?

 

Thanks!

 

 

Environment: r
Reporter: Olaf / @randomgambit
Assignee: Neal Richardson / @nealrichardson

PRs and other links:

• GitHub Pull Request #7304

_{Note: This issue was originally created as ARROW-8878. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
If you can get to github, yes, you could clone and install from the git checkout. The other alternative would be to install the arrow system packages (see http://arrow.apache.org/install/). You'll need at least libarrow-dataset-dev and libparquet-dev.

Out of curiosity, what does getOption("download.file.method") return for you?

[asfimport]

Olaf / @randomgambit:
Hi @nealrichardson, thanks for replying back. Please see below:

 

• getOption("download.file.method") returns wget

• sorry for the low-tech question, but can I install manually without cloning? That is, simply going to the github page https://github.com/apache/arrow, manually downloading the zip and then installing using the "install from zip" utility in Rstudio? Would that work correctly?

   

  Thanks!!

[asfimport]

Olaf / @randomgambit:
@nealrichardson  what do you think? interestingly, i am able to install and use the nightly version. Is the nightly package stored on another website? did you fix something in the nightly versions that might affect this?

 

thanks!

[asfimport]

Olaf / @randomgambit:
@nealrichardson  Actually I am a bit puzzled because installing from the zip does not work (I run into the same exactly errors). Is the package self-contained? Why does it need to download a couple extra files like 'distro-map.csv'? Maybe the solution would be to include them in the package files directly?

 

Thanks

[asfimport]

Neal Richardson / @nealrichardson:
You can download the zip from github and install from it. I don't know what the button in RStudio does, but you'd unzip that and in the resulting directory cd r && R CMD INSTALL .

[asfimport]

Olaf / @randomgambit:
thank you @nealrichardson  but I am still getting the same error

installing *source* package ?arrow? ...
** using staged installation
*** Successfully retrieved C++ binaries for ubuntu-18.04
Warning messages:
1: In file(file, "rt") :
 URL 'https://raw.githubusercontent.com/ursa-labs/arrow-r-nightly/master/linux/distro-map.csv': status was 'SSL connect error'
2: In unzip(bin_file, exdir = dst_dir) :
 error 1 in extracting from zip file
./configure: line 136: cd: libarrow/arrow-0.17.1.9000/lib: No such file or directory

while running install arrow still returns

 

 Warning message:
In file(file, "rt") :
 URL 'https://raw.githubusercontent.com/ursa-labs/arrow-r-nightly/master/linux/distro-map.csv': status was 'Couldn't connect to server'
 

Could these missing files be included in the zip directly? That would avoid these kind of issues. What do you think?

Thank you!!

 

 

[asfimport]

Neal Richardson / @nealrichardson:
Set LIBARROW_DOWNLOAD=false. See http://arrow.apache.org/docs/r/articles/install.html#summary-of-build-environment-variables

This wouldn't strictly be necessary but for the issue identified in #7058. I don't know why your system has set a non-standard download.file.method, but we'll have to handle that differently.

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 7304
#7304

[asfimport]

Olaf / @randomgambit:
Hi Neal, I am stilll getting the same error at the install_arrow level. I tried setting libcurl as the default file download mode but with no avail. is there a simple way to run install_arrow from R? I am not familiar with the environment variables at all.... thanks

[asfimport]

Neal Richardson / @nealrichardson:
If you've followed the installation instructions at https://arrow.apache.org/docs/r/articles/install.html and still have trouble, please open a new issue with details on how to reproduce.