Sanitzer output

[jeroen]

We have a gcc sanitizer image now, but unfortunately CMD check does not capture ASAN/UBSAN messages as it does for compiler warnings. I think Brian Ripley manually runs package tests and vignettes and greps stderr` for SAN certain messages.

So I think we might need a separate jenkins template to test for sanitizer messages.

[gaborcsardi]

We can put it in the same template I believe, and then turn it on/off via env vars, or just the image or platform name or something similar.

Do you have an outline of what we need to do?

How about just redirect the sanitizer output to a file, and then show the file?

We'll also have build artifacts at some point (built binary package, install.log output, etc.) so the sanitizer output can also be an artifact.

[jeroen]

Is there a way to capture stderr from running the tests and examples during CMD check? I am not sure exactly what CRAN does and it changes from time to time as the sanitizers get better and BDR comes up with new stuff. But I can give you an example if that is helpful.

The current CRAN version of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

[gaborcsardi]

Is there a way to capture stderr from running the tests and examples during CMD check?

I would think not. CMD check probably drops the output. But it needs a closer look.

The current release of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

test_package is not ideal, because it is testthat specific. But it is good to have an example with known sanitizer output, yes, thanks!

[jeroen]

Right I was just giving you something to see what sanitizer warnings look like.

[gaborcsardi]

Did you fix the protolite asan issues? I don't get anything from the cran version...

[gaborcsardi]

@jeroenooms forget it, I got it now....

[jeroen]

What did you end up using?

[gaborcsardi]

Oh, not done yet. But I see the SAN output at least when I run devtools::test() manually.

I think I'll just call the tests the same way as R CMD check does, some internal function if I remember correctly. And I'll run this before the regular package check.

[gaborcsardi]

Btw. do you think it is OK not to run R CMD check at all, in the SAN image? Just thinking. People would use this image to get the SAN output, and they don't care about R CMD check much, I guess.

[gaborcsardi]

It seems that

RD CMD INSTALL <pkg-tar-gz>
tar xzf <pkg-tar-gz>
cd <pkg>/tests
RD -e 'tools:::.runPackageTestsR()'
cat *.Rout*

will do the job. Or sg better than the last cat, to show the actual filenames as well.

[jeroen]

Actually cran also tests examples and vignettes, and then grep's for sanitizer output lines. See e.g. https://cran.r-project.org/web/checks/check_results_mongolite.html

[gaborcsardi]

@jeroenooms Thanks, good to know. I can easily do that, too. And I also have symbolize....

[gaborcsardi]

Slowly getting there. I'll start with running this:

## Install the package, also decompress to run the tests
RD CMD INSTALL "$1"
tar xzf "$1"
pkg=$(echo $1 | sed 's/_.*$//')

## Tests
(
  echo "Running tests"
  cd ${pkg}/tests
  RD -e 'tools:::.runPackageTestsR()'
  cat *.Rout*
)

## Examples
echo "Running examples"
RD -q -e "tools:::.createExdotR('$pkg', system.file(package = '$pkg'))"
RD --slave -f ${pkg}-Ex.R

## Vignettes
echo "Running vignette code"
if [[ -e "${pkg}/vignettes" ]]; then
  RD -q -e "tools::buildVignettes(dir = '.', tangle = TRUE)"
  find ${pkg}/vignettes -name "*.R" -maxdepth 1 -print -exec RD -q -f \{\} \;
fi

[gaborcsardi]

Another question how to trigger these checks. I suppose I can

• put the san.sh file (with contents as above) into the image
• define an RHUB_CHECK_COMMAND variable, which calls san.sh in the SAN image, and this is run instead of R CMD check.
• fall back to R CMD CHECK if RHUB_CHECK_COMMAND is not defined.

[jeroen]

Yeah I think that makes sense. R CMD check itself for UBSAN is not super interesting, it's really just the output.

[gaborcsardi]

OK, I think this is done. E.g. https://builder.r-hub.io/status/protolite_1.5.tar.gz-36a10729c4af447894c8d7f9f67ed4f7

It also looks good from the terminal...