Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanitzer output #10

Closed
jeroen opened this issue Sep 6, 2016 · 16 comments
Closed

Sanitzer output #10

jeroen opened this issue Sep 6, 2016 · 16 comments

Comments

@jeroen
Copy link
Member

@jeroen jeroen commented Sep 6, 2016

We have a gcc sanitizer image now, but unfortunately CMD check does not capture ASAN/UBSAN messages as it does for compiler warnings. I think Brian Ripley manually runs package tests and vignettes and greps stderr` for SAN certain messages.

So I think we might need a separate jenkins template to test for sanitizer messages.

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 6, 2016

We can put it in the same template I believe, and then turn it on/off via env vars, or just the image or platform name or something similar.

Do you have an outline of what we need to do?

How about just redirect the sanitizer output to a file, and then show the file?

We'll also have build artifacts at some point (built binary package, install.log output, etc.) so the sanitizer output can also be an artifact.

@jeroen
Copy link
Member Author

@jeroen jeroen commented Sep 6, 2016

Is there a way to capture stderr from running the tests and examples during CMD check? I am not sure exactly what CRAN does and it changes from time to time as the sanitizers get better and BDR comes up with new stuff. But I can give you an example if that is helpful.

The current CRAN version of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 6, 2016

Is there a way to capture stderr from running the tests and examples during CMD check?

I would think not. CMD check probably drops the output. But it needs a closer look.

The current release of the protolite package will display a ton of UBSAN warnings during test_package("protolite")

test_package is not ideal, because it is testthat specific. But it is good to have an example with known sanitizer output, yes, thanks!

@jeroen
Copy link
Member Author

@jeroen jeroen commented Sep 6, 2016

Right I was just giving you something to see what sanitizer warnings look like.

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 22, 2016

Did you fix the protolite asan issues? I don't get anything from the cran version...

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 22, 2016

@jeroenooms forget it, I got it now....

@jeroen
Copy link
Member Author

@jeroen jeroen commented Sep 22, 2016

What did you end up using?

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 22, 2016

Oh, not done yet. But I see the SAN output at least when I run devtools::test() manually.

I think I'll just call the tests the same way as R CMD check does, some internal function if I remember correctly. And I'll run this before the regular package check.

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 22, 2016

Btw. do you think it is OK not to run R CMD check at all, in the SAN image? Just thinking. People would use this image to get the SAN output, and they don't care about R CMD check much, I guess.

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 22, 2016

It seems that

RD CMD INSTALL <pkg-tar-gz>
tar xzf <pkg-tar-gz>
cd <pkg>/tests
RD -e 'tools:::.runPackageTestsR()'
cat *.Rout*

will do the job. Or sg better than the last cat, to show the actual filenames as well.

@jeroen
Copy link
Member Author

@jeroen jeroen commented Sep 22, 2016

Actually cran also tests examples and vignettes, and then grep's for sanitizer output lines. See e.g. https://cran.r-project.org/web/checks/check_results_mongolite.html

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 22, 2016

@jeroenooms Thanks, good to know. I can easily do that, too. And I also have symbolize....

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 27, 2016

Slowly getting there. I'll start with running this:

## Install the package, also decompress to run the tests
RD CMD INSTALL "$1"
tar xzf "$1"
pkg=$(echo $1 | sed 's/_.*$//')

## Tests
(
  echo "Running tests"
  cd ${pkg}/tests
  RD -e 'tools:::.runPackageTestsR()'
  cat *.Rout*
)

## Examples
echo "Running examples"
RD -q -e "tools:::.createExdotR('$pkg', system.file(package = '$pkg'))"
RD --slave -f ${pkg}-Ex.R

## Vignettes
echo "Running vignette code"
if [[ -e "${pkg}/vignettes" ]]; then
  RD -q -e "tools::buildVignettes(dir = '.', tangle = TRUE)"
  find ${pkg}/vignettes -name "*.R" -maxdepth 1 -print -exec RD -q -f \{\} \;
fi
@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 27, 2016

Another question how to trigger these checks. I suppose I can

  • put the san.sh file (with contents as above) into the image
  • define an RHUB_CHECK_COMMAND variable, which calls san.sh in the SAN image, and this is run instead of R CMD check.
  • fall back to R CMD CHECK if RHUB_CHECK_COMMAND is not defined.
@jeroen
Copy link
Member Author

@jeroen jeroen commented Sep 27, 2016

Yeah I think that makes sense. R CMD check itself for UBSAN is not super interesting, it's really just the output.

@gaborcsardi
Copy link
Contributor

@gaborcsardi gaborcsardi commented Sep 27, 2016

OK, I think this is done. E.g. https://builder.r-hub.io/status/protolite_1.5.tar.gz-36a10729c4af447894c8d7f9f67ed4f7

It also looks good from the terminal...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.