Rewrite no_init() and other functions to reduce false positives #893

krlmlr · 2018-08-18T08:17:19Z

Reference: #892.

codecov-io · 2018-08-18T08:31:08Z

Codecov Report

Merging #893 into master will increase coverage by <.01%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master     #893      +/-   ##
==========================================
+ Coverage   90.14%   90.15%   +<.01%     
==========================================
  Files          71       71              
  Lines        3279     3282       +3     
==========================================
+ Hits         2956     2959       +3     
  Misses        323      323

Impacted Files	Coverage Δ
inst/include/Rcpp/exceptions.h	`7.69% <ø> (ø)`	⬆️
inst/include/Rcpp/protection/Shelter.h	`0% <0%> (ø)`	⬆️
inst/include/Rcpp/api/meat/Rcpp_eval.h	`66.66% <100%> (ø)`	⬆️
inst/include/Rcpp/protection/Shield.h	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 522e66d...b735942. Read the comment docs.

eddelbuettel · 2018-08-18T12:52:54Z

inst/include/Rcpp/protection/Shield.h

@@ -25,12 +25,16 @@ namespace Rcpp{
        return x ;
    }

+    inline SEXP Rcpp_unprotect(int i){


This function wants to be void or else g++ yells at you.

You can cherry-pick it from 0df99c8

Thanks, done.

kevinushey · 2018-08-18T16:02:17Z

inst/include/Rcpp/vector/no_init.h

-    operator Vector<RTYPE, StoragePolicy>() const {
-        return Rf_allocVector(RTYPE, size) ;
+    template <int RTYPE>
+    operator Vector<RTYPE, PreserveStorage>() const {


Why change the template parameters here? Do we need separate operators for each of the storage policies?

I think no_init() should always return a properly protected vector, that's why I removed this template parameter. But I'm not sure.

The intention is that the caller can indicate whether they want protection or want -- by default, they get a protected vector; if they don't want one then they can request conversion to the NoPreserveStorage variant.

IIUC, since this function was previously returning a raw SEXP, the associated constructor would be called on that return value and that constructor would take care of preserving the underlying storage when required.

As I recall, the "feature" of no_init() was to save the handful of cycles needed to set the memory to zero, but not to alter anything on protection or lack thereof.

no_init() effectively returns an uninitialized vector (via constructing a temporary and calling operator Vector<...>() on it). To be on the very safe side, this uninitialized vector needs to be protected during its lifetime; the protection state of the target is of no relevance. This corner case is perhaps irrelevant.

Vector::Vector(SEXP) seems to coerce (in the very generic case, not affecting us, but may be hard to tell for rchk). Maybe rchk thinks (or sees) that this constructor is involved in Vector<...> x = no_init(...)?

Happy to revert to the original implementation (with the second template parameter), but I'll check first if rchk eats this.

kevinushey · 2018-08-18T16:05:06Z

My understanding is that rchk was primarily designed to diagnose C rather than C++ code. I would prefer not making these kinds of changes just to appease the tool unless we really have a good reason.

krlmlr · 2018-08-19T09:23:21Z

The proposed changes reduce false positives reported by rchk not only in Rcpp, but in packages that merely use Rcpp::no_init() (directly or indirectly). CRAN uses rchk, fixing these issues will help them. (We've been asked about these when submitting dplyr 0.7.6, we could resolve many but not all on the client side by using the second syntax variant in #892.) I added comments to explain the changes.

The rchk documentation recommends rewriting code to reduce the number of false positives too.

Code that relies entirely on Rcpp's vector classes doesn't benefit much from rchk. On the other hand, ideally such code shouldn't produce as few false positives as possible, ideally none.

eddelbuettel · 2018-08-19T13:21:06Z

A quick check at GitHub searching 'user:cran no_init' shows that there are about a handful of packages using no_init() so it may be worth it.

A reverse-depends run seems to suggest no new issues (package 'vapour' came up failing but I also see the same issue for macOS at its CRAN checks so this may not be us).

What about no_init.h though and Kevin's comment? Looking at the file I see two cases for vector and matrix. You addressed only one. Should probably be both. But then again, why not keep Rf_allocVector() and Rf_allocMatrix() ?

krlmlr · 2018-08-19T19:15:15Z

Thanks. I'll address the matrix case and also try to simplify the code more.

I still think we need to return an object with protection, if we keep the StoragePolicy template argument we might end up with unprotected objects again.

I don't understand "keep Rf_allocVector()": the code still uses Rf_allocVector() but just protects it explicitly.

kevinushey · 2018-08-20T15:29:04Z

We've been asked about these when submitting dplyr 0.7.6

Since CRAN is enforcing these checks I agree we should do what we can to reduce false positives.

kevinushey · 2018-08-20T15:32:37Z

inst/include/Rcpp/exceptions.h

@@ -243,7 +243,7 @@ namespace internal {
    inline bool is_Rcpp_eval_call(SEXP expr) {
        SEXP sys_calls_symbol = Rf_install("sys.calls");
        SEXP identity_symbol = Rf_install("identity");
-        SEXP identity_fun = Rf_findFun(identity_symbol, R_BaseEnv);
+        Shield<SEXP> identity_fun(Rf_findFun(identity_symbol, R_BaseEnv));
        SEXP tryCatch_symbol = Rf_install("tryCatch");


Does rchk warn about the unprotected use of 'install' here?

Yes: krlmlr/RcppNoInitTest@38aa117#diff-7f4f9d499ac2cdf819002146b21b8e18L24

So should the PR cover that too then?

I was confused. No, rchk knows that the Rf_install() result is protected already, but Rf_install() could be an allocation (in theory) which could clean up the result of Rf_findFun().

kevinushey · 2018-08-20T15:33:30Z

inst/include/Rcpp/vector/no_init.h

+    operator Vector<RTYPE, PreserveStorage>() const {
+        // Explicitly protect temporary vector to avoid false positive
+        // with rchk (#892)
+        SEXP x = PROTECT(Rf_allocVector(RTYPE, size));


What about using Shield<SEXP> here rather than raw protect / unprotect?

Need to check if rchk is satisfied.

eddelbuettel · 2018-08-26T11:22:27Z

Thanks for coming back to this -- looks pretty good now (though it is early here and the coffee may not have settled ... ;-)

One thing still missing is the Matrix case on no_init.h. Could you add that?

krlmlr · 2018-08-26T12:29:39Z

Thanks for the heads-up, I forgot. (Need to wait for the checks.)

eddelbuettel · 2018-08-26T12:32:39Z

Err, don't you test locally before sending a commit ?

krlmlr · 2018-08-26T12:49:27Z

Guilty as charged.

eddelbuettel · 2018-08-26T12:51:07Z

I may not be entirely innocent of the practice either but I hate the red outcomes with such a vengeance that I try avoid this. Nice use of git commit --amend though :)

eddelbuettel · 2018-08-26T12:58:50Z

I will set up another reverse-depends check. Not expecting any trouble. Should be ready to merge.

@kevinushey @jjallaire @thirdwing @nathan-russell and anybody else: Comments? Concerns?

eddelbuettel · 2018-08-26T13:09:49Z

I canceled and restarted that one, and (of course) just after I canceled it completed. It had exit-ed fine with value of zero but was hanging there...

krlmlr · 2018-08-26T13:25:18Z

Thanks. Looks super-green now.

eddelbuettel · 2018-08-26T21:03:11Z

No new issue in level-one reverse depends check, results at RcppCore/rcpp-logs@1d9d5fa

krlmlr added 2 commits August 18, 2018 09:51

Explicitly protect temporary vector in no_init()

5dbfca1

Fix more false rchk positives

4c4721a

eddelbuettel reviewed Aug 18, 2018

View reviewed changes

kevinushey reviewed Aug 18, 2018

View reviewed changes

eddelbuettel and others added 2 commits August 19, 2018 11:04

make Rcpp_unprotect a void function

3fa6ca0

Add comments

72f4db1

kevinushey reviewed Aug 20, 2018

View reviewed changes

Tweaks

3d5e62f

Support matrix case

b735942

krlmlr changed the title ~~WIP: rewrite no_init() and other functions to reduce false positives~~ Rewrite no_init() and other functions to reduce false positives Aug 26, 2018

eddelbuettel approved these changes Aug 26, 2018

View reviewed changes

kevinushey approved these changes Aug 27, 2018

View reviewed changes

eddelbuettel merged commit 4813640 into RcppCore:master Aug 27, 2018

krlmlr deleted the f-no-init-protect branch September 15, 2018 09:06

krlmlr mentioned this pull request Sep 18, 2018

Redesign of hybrid and standard evaluation tidyverse/dplyr#3799

Merged

eddelbuettel mentioned this pull request Sep 20, 2018

no_init() triggers rchk errors #892

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite no_init() and other functions to reduce false positives #893

Rewrite no_init() and other functions to reduce false positives #893

krlmlr commented Aug 18, 2018

codecov-io commented Aug 18, 2018 •

edited

Loading

eddelbuettel Aug 18, 2018

eddelbuettel Aug 18, 2018

krlmlr Aug 19, 2018

kevinushey Aug 18, 2018

krlmlr Aug 19, 2018

kevinushey Aug 20, 2018

eddelbuettel Aug 20, 2018

krlmlr Aug 21, 2018

kevinushey commented Aug 18, 2018

krlmlr commented Aug 19, 2018

eddelbuettel commented Aug 19, 2018

krlmlr commented Aug 19, 2018

kevinushey commented Aug 20, 2018

kevinushey Aug 20, 2018

krlmlr Aug 21, 2018

eddelbuettel Aug 21, 2018

krlmlr Aug 21, 2018

kevinushey Aug 20, 2018

krlmlr Aug 21, 2018

eddelbuettel commented Aug 26, 2018

krlmlr commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

krlmlr commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

krlmlr commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

Rewrite no_init() and other functions to reduce false positives #893

Rewrite no_init() and other functions to reduce false positives #893

Conversation

krlmlr commented Aug 18, 2018

codecov-io commented Aug 18, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinushey commented Aug 18, 2018

krlmlr commented Aug 19, 2018

eddelbuettel commented Aug 19, 2018

krlmlr commented Aug 19, 2018

kevinushey commented Aug 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddelbuettel commented Aug 26, 2018

krlmlr commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

krlmlr commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

krlmlr commented Aug 26, 2018

eddelbuettel commented Aug 26, 2018

codecov-io commented Aug 18, 2018 •

edited

Loading