Submodules are not updated when R package is in a different directory to .gitmodules file

[awalker89]

The xgboost example from the README fails because the submodules are not updated.

# remotes::install_git("https://github.com/dmlc/xgboost/", subdir = "R-package", quiet = FALSE)
remotes::install_github("dmlc/xgboost/R-package")
* installing *source* package 'xgboost' ...
** libs

*** arch - i386
Makevars.win:29: -DXGBOOST_STRICT_R_MODE=1
Makevars.win:29: -DDMLC_LOG_BEFORE_THROW=0
Makevars.win:29: -DDMLC_ENABLE_STD_THREAD=0
Makevars.win:29: -DDMLC_DISABLE_STDIN=1
Makevars.win:29: -DDMLC_LOG_CUSTOMIZE=1
Makevars.win:29: -DXGBOOST_CUSTOMIZE_LOGGER=1
Makevars.win:29: -DRABIT_CUSTOMIZE_MSG_
Makevars.win:29: -DRABIT_STRICT_CXX98_
Makevars.win:29: -DDMLC_CXX11_THREAD_LOCAL=0
cp -r ../../src .
cp: cannot stat '../../src': No such file or directory
make: *** [xgblib] Error 1
Warning: running command 'make -f "Makevars.win" -f "C:/PROGRA~1/R/R-34~1.1/etc/i386/Makeconf" -f "C:/PROGRA~1/R/R-34~1.1/share/make/winshlib.mk" CXX='$(CXX11) $(CXX11STD)' CXXFLAGS='$(CXX11FLAGS)' CXXPICFLAGS='$(CXX11PICFLAGS)' SHLIB_LDFLAGS='$(SHLIB_CXX11LDFLAGS)' SHLIB_LD='$(SHLIB_CXX11LD)' SHLIB="xgboost.dll" ' had status 2
ERROR: compilation failed for package 'xgboost'

The .gitmodules file is not being found because it lives in the parent directory of the "R-package" subdirectory.

In this situation (for xgboost) the .gitmodules file is in the bundle directory and this:

remotes/R/install-remote.R

Lines 52 to 55 in 0ae9774

	source <- source_pkg(bundle, subdir = remote$subdir)
	on.exit(unlink(source, recursive = TRUE), add = TRUE)
	update_submodules(source, quiet)

needs to be

update_submodules(bundle, quiet)

My hacky fix was to look for the .gitmodules in the parent directory of source if not found.
source is also modified so the modules are updated in the correct directory.

remotes/R/submodule.R

Lines 88 to 91 in 97bbf81

	file <- file.path(source, ".gitmodules")
	if (!file.exists(file)) {
	return()
	}

becomes:

file <- file.path(source, ".gitmodules")
if (!file.exists(file)) {
  source <- file.path(source, "..")
  file <- file.path(source, ".gitmodules")
  if (!file.exists(file)) {
    return()
  }
}

Then this works:

> remotes::install_git("https://github.com/dmlc/xgboost/", subdir = "R-package", quiet = FALSE, repos = NULL, build = FALSE)
Downloading git repo https://github.com/dmlc/xgboost/
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/dmlc/dmlc-core /tmp/RtmpBk9OdU/file8dc61d3b0618/R-package/../dmlc-core
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/dmlc/rabit /tmp/RtmpBk9OdU/file8dc61d3b0618/R-package/../rabit
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/NVlabs/cub /tmp/RtmpBk9OdU/file8dc61d3b0618/R-package/../cub
Skipping 3 packages not available: data.table, magrittr, stringi
Installing package into ‘/home/monitor/R/x86_64-redhat-linux-gnu-library/3.5’
(as ‘lib’ is unspecified)
* installing *source* package ‘xgboost’ ...
configure: creating ./config.status
config.status: creating src/Makevars
** libs
Makevars:17: -DXGBOOST_STRICT_R_MODE=1
Makevars:17: -DDMLC_LOG_BEFORE_THROW=0
Makevars:17: -DDMLC_ENABLE_STD_THREAD=1
Makevars:17: -DDMLC_DISABLE_STDIN=1
Makevars:17: -DDMLC_LOG_CUSTOMIZE=1
Makevars:17: -DXGBOOST_CUSTOMIZE_LOGGER=1
Makevars:17: -DRABIT_CUSTOMIZE_MSG_
Makevars:17: -DRABIT_STRICT_CXX98_
g++ -m64 -std=gnu++11 -I"/usr/include/R" -DNDEBUG -I../..//include -I../..//dmlc-core/include -I../..//rabit/include -I../../ -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -I/usr/local/include  -fopenmp  -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -c xgboost_R.cc -o xgboost_R.o
g++ -m64 -std=gnu++11 -I"/usr/include/R" -DNDEBUG -I../..//include -I../..//dmlc-core/include -I../..//rabit/include -I../../ -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -I/usr/local/include  -fopenmp  -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -c xgboost_custom.cc -o xgboost_custom.o
gcc -m64 -std=gnu99 -I"/usr/include/R" -DNDEBUG -I../..//include -I../..//dmlc-core/include -I../..//rabit/include -I../../ -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -I/usr/local/include   -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic  -c xgboost_assert.c -o xgboost_assert.o
gcc -m64 -std=gnu99 -I"/usr/include/R" -DNDEBUG -I../..//include -I../..//dmlc-core/include -I../..//rabit/include -I../../ -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -I/usr/local/include   -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic  -c init.c -o init.o
g++ -m64 -std=gnu++11 -I"/usr/include/R" -DNDEBUG -I../..//include -I../..//dmlc-core/include -I../..//rabit/include -I../../ -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_  -I/usr/local/include  -fopenmp  -fpic  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -c ../..//amalgamation/xgboost-all0.cc -o ../..//amalgamation/xgboost-all0.o
In file included from ../..//amalgamation/../src/data/./sparse_page_writer.h:20:0,
                 from ../..//amalgamation/../src/data/data.cc:9,
                 from ../..//amalgamation/xgboost-all0.cc:31:
../..//dmlc-core/include/dmlc/concurrency.h:31:0: warning: ignoring #pragma clang diagnostic [-Wunknown-pragmas]
 #pragma clang diagnostic push
 ^
../..//dmlc-core/include/dmlc/concurrency.h:32:0: warning: ignoring #pragma clang diagnostic [-Wunknown-pragmas]
 #pragma clang diagnostic ignored "-Wbraced-scalar-init"
 ^
../..//dmlc-core/include/dmlc/concurrency.h:35:0: warning: ignoring #pragma clang diagnostic [-Wunknown-pragmas]
.
.
.

Session Info

Using the latest version from `remotes::install_github("r-lib/remotes", force = TRUE)`

> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252    LC_MONETARY=English_Australia.1252 LC_NUMERIC=C                      
[5] LC_TIME=English_Australia.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] ps_1.2.0          prettyunits_1.0.2 rprojroot_1.3-2   withr_2.1.2       crayon_1.3.4      assertthat_0.2.0  R6_2.3.0          backports_1.1.2   git2r_0.23.0     
[10] magrittr_1.5      debugme_1.1.0     cli_1.0.1         curl_3.2          remotes_2.0.2     rstudioapi_0.8    callr_3.0.0       tools_3.4.1       yaml_2.2.0       
[19] compiler_3.4.1    processx_3.2.0    base64enc_0.1-3   pkgbuild_1.0.2   

[pommedeterresautee]

I have implemented the trick above but it doesn't work.
The submodules are found, and cloned, but the installation through remote fails because a file to include (from a submodule) is not found!!!
On the same computer, I can install XGBoost R package manually (so no dep missing).

> remotes::install_github("dmlc/xgboost/R-package")
Downloading GitHub repo dmlc/xgboost@master
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/dmlc/dmlc-core /tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/../dmlc-core
Clonage dans '/tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/../dmlc-core'...
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/dmlc/rabit /tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/../rabit
Clonage dans '/tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/../rabit'...
'/usr/bin/git' clone --depth 1 --no-hardlinks --recurse-submodules https://github.com/NVlabs/cub /tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/../cub
Clonage dans '/tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/../cub'...
✔  checking for file ‘/tmp/Rtmp3p3kcp/remotesee15e5dbae6/dmlc-xgboost-9b9e298/R-package/DESCRIPTION’ ...
─  preparing ‘xgboost’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  looking to see if a ‘data/datalist’ file should be added
─  building ‘xgboost_1.0.0.1.tar.gz’
   
Installing package into ‘/home/geantvert/R/x86_64-pc-linux-gnu-library/3.5’
(as ‘lib’ is unspecified)
* installing *source* package ‘xgboost’ ...
configure: creating ./config.status
config.status: creating src/Makevars
** libs
Makevars:17: -DXGBOOST_STRICT_R_MODE=1
Makevars:17: -DDMLC_LOG_BEFORE_THROW=0
Makevars:17: -DDMLC_ENABLE_STD_THREAD=1
Makevars:17: -DDMLC_DISABLE_STDIN=1
Makevars:17: -DDMLC_LOG_CUSTOMIZE=1
Makevars:17: -DXGBOOST_CUSTOMIZE_LOGGER=1
Makevars:17: -DRABIT_CUSTOMIZE_MSG_
Makevars:17: -DRABIT_STRICT_CXX98_
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../..//include -I../..//dmlc-core/include -I../..//rabit/include -I../../ -DXGBOOST_STRICT_R_MODE=1 -DDMLC_LOG_BEFORE_THROW=0 -DDMLC_ENABLE_STD_THREAD=1 -DDMLC_DISABLE_STDIN=1 -DDMLC_LOG_CUSTOMIZE=1 -DXGBOOST_CUSTOMIZE_LOGGER=1 -DRABIT_CUSTOMIZE_MSG_ -DRABIT_STRICT_CXX98_    -fopenmp -pthread -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-el7SHG/r-base-3.5.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c xgboost_R.cc -o xgboost_R.o
xgboost_R.cc:2:10: fatal error: dmlc/logging.h: Aucun fichier ou dossier de ce type
 #include <dmlc/logging.h>
          ^~~~~~~~~~~~~~~~
compilation terminated.
make: *** [/usr/lib/R/etc/Makeconf:169: xgboost_R.o] Error 1
ERROR: compilation failed for package ‘xgboost’
* removing ‘/home/geantvert/R/x86_64-pc-linux-gnu-library/3.5/xgboost’
* restoring previous ‘/home/geantvert/R/x86_64-pc-linux-gnu-library/3.5/xgboost’
 Erreur : Failed to install 'xgboost' from GitHub:
  (converti depuis l'avis) installation of package ‘/tmp/Rtmp3p3kcp/fileee13629f27d/xgboost_1.0.0.1.tar.gz’ had non-zero exit status