New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanitzer output #10

Closed
jeroen opened this Issue Sep 6, 2016 · 16 comments

Comments

Projects
None yet
2 participants
@jeroen
Member

jeroen commented Sep 6, 2016

We have a gcc sanitizer image now, but unfortunately CMD check does not capture ASAN/UBSAN messages as it does for compiler warnings. I think Brian Ripley manually runs package tests and vignettes and greps stderr` for SAN certain messages.

So I think we might need a separate jenkins template to test for sanitizer messages.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 6, 2016

Member

We can put it in the same template I believe, and then turn it on/off via env vars, or just the image or platform name or something similar.

Do you have an outline of what we need to do?

How about just redirect the sanitizer output to a file, and then show the file?

We'll also have build artifacts at some point (built binary package, install.log output, etc.) so the sanitizer output can also be an artifact.

Member

gaborcsardi commented Sep 6, 2016

We can put it in the same template I believe, and then turn it on/off via env vars, or just the image or platform name or something similar.

Do you have an outline of what we need to do?

How about just redirect the sanitizer output to a file, and then show the file?

We'll also have build artifacts at some point (built binary package, install.log output, etc.) so the sanitizer output can also be an artifact.

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 6, 2016

Member

Is there a way to capture stderr from running the tests and examples during CMD check? I am not sure exactly what CRAN does and it changes from time to time as the sanitizers get better and BDR comes up with new stuff. But I can give you an example if that is helpful.

The current CRAN version of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

Member

jeroen commented Sep 6, 2016

Is there a way to capture stderr from running the tests and examples during CMD check? I am not sure exactly what CRAN does and it changes from time to time as the sanitizers get better and BDR comes up with new stuff. But I can give you an example if that is helpful.

The current CRAN version of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 6, 2016

Member

Is there a way to capture stderr from running the tests and examples during CMD check?

I would think not. CMD check probably drops the output. But it needs a closer look.

The current release of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

test_package is not ideal, because it is testthat specific. But it is good to have an example with known sanitizer output, yes, thanks!

Member

gaborcsardi commented Sep 6, 2016

Is there a way to capture stderr from running the tests and examples during CMD check?

I would think not. CMD check probably drops the output. But it needs a closer look.

The current release of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

test_package is not ideal, because it is testthat specific. But it is good to have an example with known sanitizer output, yes, thanks!

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 6, 2016

Member

Right I was just giving you something to see what sanitizer warnings look like.

Member

jeroen commented Sep 6, 2016

Right I was just giving you something to see what sanitizer warnings look like.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 22, 2016

Member

Did you fix the protolite asan issues? I don't get anything from the cran version...

Member

gaborcsardi commented Sep 22, 2016

Did you fix the protolite asan issues? I don't get anything from the cran version...

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 22, 2016

Member

@jeroenooms forget it, I got it now....

Member

gaborcsardi commented Sep 22, 2016

@jeroenooms forget it, I got it now....

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 22, 2016

Member

What did you end up using?

Member

jeroen commented Sep 22, 2016

What did you end up using?

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 22, 2016

Member

Oh, not done yet. But I see the SAN output at least when I run devtools::test() manually.

I think I'll just call the tests the same way as R CMD check does, some internal function if I remember correctly. And I'll run this before the regular package check.

Member

gaborcsardi commented Sep 22, 2016

Oh, not done yet. But I see the SAN output at least when I run devtools::test() manually.

I think I'll just call the tests the same way as R CMD check does, some internal function if I remember correctly. And I'll run this before the regular package check.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 22, 2016

Member

Btw. do you think it is OK not to run R CMD check at all, in the SAN image? Just thinking. People would use this image to get the SAN output, and they don't care about R CMD check much, I guess.

Member

gaborcsardi commented Sep 22, 2016

Btw. do you think it is OK not to run R CMD check at all, in the SAN image? Just thinking. People would use this image to get the SAN output, and they don't care about R CMD check much, I guess.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 22, 2016

Member

It seems that

RD CMD INSTALL <pkg-tar-gz>
tar xzf <pkg-tar-gz>
cd <pkg>/tests
RD -e 'tools:::.runPackageTestsR()'
cat *.Rout*

will do the job. Or sg better than the last cat, to show the actual filenames as well.

Member

gaborcsardi commented Sep 22, 2016

It seems that

RD CMD INSTALL <pkg-tar-gz>
tar xzf <pkg-tar-gz>
cd <pkg>/tests
RD -e 'tools:::.runPackageTestsR()'
cat *.Rout*

will do the job. Or sg better than the last cat, to show the actual filenames as well.

@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 22, 2016

Member

Actually cran also tests examples and vignettes, and then grep's for sanitizer output lines. See e.g. https://cran.r-project.org/web/checks/check_results_mongolite.html

Member

jeroen commented Sep 22, 2016

Actually cran also tests examples and vignettes, and then grep's for sanitizer output lines. See e.g. https://cran.r-project.org/web/checks/check_results_mongolite.html

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 22, 2016

Member

@jeroenooms Thanks, good to know. I can easily do that, too. And I also have symbolize....

Member

gaborcsardi commented Sep 22, 2016

@jeroenooms Thanks, good to know. I can easily do that, too. And I also have symbolize....

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 27, 2016

Member

Slowly getting there. I'll start with running this:

## Install the package, also decompress to run the tests
RD CMD INSTALL "$1"
tar xzf "$1"
pkg=$(echo $1 | sed 's/_.*$//')

## Tests
(
  echo "Running tests"
  cd ${pkg}/tests
  RD -e 'tools:::.runPackageTestsR()'
  cat *.Rout*
)

## Examples
echo "Running examples"
RD -q -e "tools:::.createExdotR('$pkg', system.file(package = '$pkg'))"
RD --slave -f ${pkg}-Ex.R

## Vignettes
echo "Running vignette code"
if [[ -e "${pkg}/vignettes" ]]; then
  RD -q -e "tools::buildVignettes(dir = '.', tangle = TRUE)"
  find ${pkg}/vignettes -name "*.R" -maxdepth 1 -print -exec RD -q -f \{\} \;
fi
Member

gaborcsardi commented Sep 27, 2016

Slowly getting there. I'll start with running this:

## Install the package, also decompress to run the tests
RD CMD INSTALL "$1"
tar xzf "$1"
pkg=$(echo $1 | sed 's/_.*$//')

## Tests
(
  echo "Running tests"
  cd ${pkg}/tests
  RD -e 'tools:::.runPackageTestsR()'
  cat *.Rout*
)

## Examples
echo "Running examples"
RD -q -e "tools:::.createExdotR('$pkg', system.file(package = '$pkg'))"
RD --slave -f ${pkg}-Ex.R

## Vignettes
echo "Running vignette code"
if [[ -e "${pkg}/vignettes" ]]; then
  RD -q -e "tools::buildVignettes(dir = '.', tangle = TRUE)"
  find ${pkg}/vignettes -name "*.R" -maxdepth 1 -print -exec RD -q -f \{\} \;
fi
@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 27, 2016

Member

Another question how to trigger these checks. I suppose I can

  • put the san.sh file (with contents as above) into the image
  • define an RHUB_CHECK_COMMAND variable, which calls san.sh in the SAN image, and this is run instead of R CMD check.
  • fall back to R CMD CHECK if RHUB_CHECK_COMMAND is not defined.
Member

gaborcsardi commented Sep 27, 2016

Another question how to trigger these checks. I suppose I can

  • put the san.sh file (with contents as above) into the image
  • define an RHUB_CHECK_COMMAND variable, which calls san.sh in the SAN image, and this is run instead of R CMD check.
  • fall back to R CMD CHECK if RHUB_CHECK_COMMAND is not defined.
@jeroen

This comment has been minimized.

Show comment
Hide comment
@jeroen

jeroen Sep 27, 2016

Member

Yeah I think that makes sense. R CMD check itself for UBSAN is not super interesting, it's really just the output.

Member

jeroen commented Sep 27, 2016

Yeah I think that makes sense. R CMD check itself for UBSAN is not super interesting, it's really just the output.

@gaborcsardi

This comment has been minimized.

Show comment
Hide comment
@gaborcsardi

gaborcsardi Sep 27, 2016

Member

OK, I think this is done. E.g. https://builder.r-hub.io/status/protolite_1.5.tar.gz-36a10729c4af447894c8d7f9f67ed4f7

It also looks good from the terminal...

Member

gaborcsardi commented Sep 27, 2016

OK, I think this is done. E.g. https://builder.r-hub.io/status/protolite_1.5.tar.gz-36a10729c4af447894c8d7f9f67ed4f7

It also looks good from the terminal...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment