[ci] [R-package] incorrect R version being used on some R-package CI jobs

[jameslamb]

Description

At least on Linux, r-package CI jobs in this project appear to be using the wrong R version.

For example, on the most recent r-package (ubuntu-22.04, gcc, R 3.6, cmake build on master (build link), I see the following in the logs.

* using R version 4.2.2 Patched (2022-11-10 r83330)
* using platform: x86_64-pc-linux-gnu (64-bit)

and

R CMD check build logs:
...
installing via 'install.libs.R' to /home/runner/work/LightGBM/LightGBM/lightgbm.Rcheck/00LOCK-lightgbm/00new/lightgbm
-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
...
-- R version passed into FindLibR.cmake: 4.2.2

This means that at least for some combinations of operating system, compiler, and build type, the R package is not actually being tested against R 3.6.

Reproducible example

I noticed this on ubuntu-22.04 (ubuntu-latest) Linux R 3.6 jobs, but it might be happening on others.

Environment info

At least latest master (a174893), unsure how long this has been going on.

Additional Comments

I suspect that this is happening because of a combination of the following:

• these jobs run on GitHub Actions VMs, not in containers, and those VMs have a LOT of stuff pre-installed, including R
  • see the READMEs at https://github.com/actions/runner-images/tree/main/images/linux
• CI scripts like setup.sh and test_r_package.sh don't use set -e, so if commands installing stuff fail and don't explicitly exit with a non-0 code, scripts will continue running
  • this is intentional, see [ci] removed need to explicitly exit from failed commands in CI #2497 (comment)

At least for Linux jobs, I hope that #5638 will prevent this "silently fell back to software from GitHub Actions" issue.

To close this issue, CI checks should be added which explicitly confirm that the expected version of R was used, similar to

LightGBM/.ci/test_r_package.sh

Lines 256 to 272 in a174893

	# this check makes sure that CI builds of the package
	# actually use MM_MALLOC preprocessor definition
	if [[ $R_BUILD_TYPE == "cran" ]]; then
	mm_malloc_working=$(
	cat $BUILD_LOG_FILE \
	| grep --count -E "checking whether MM_MALLOC work.*yes"
	)
	else
	mm_malloc_working=$(
	cat $BUILD_LOG_FILE \
	| grep --count -E ".*Performing Test MM_MALLOC - Success"
	)
	fi
	if [[ $mm_malloc_working -ne 1 ]]; then
	echo "MM_MALLOC test was not passed"
	exit -1
	fi

I remember @StrikerRUS recommended that in the past, but I can't find the comment to link to.

[jameslamb]

The correct version is being used on macOS and Linux CI jobs, per this:

LightGBM/.ci/test_r_package.sh

Lines 234 to 241 in 646dfed

	used_correct_r_version=$(
	cat $LOG_FILE_NAME \
	| grep --count "using R version ${R_VERSION}"
	)
	if [[ $used_correct_r_version -ne 1 ]]; then
	echo "Unexpected R version was used. Expected '${R_VERSION}'."
	exit -1
	fi

So only Windows checks need to still be added. I'll put up a PR for that shortly.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.