Simplify hybrid evaluation

[krlmlr]

Fixes #912.

Fixes #1012 (but untestable because of hadley/lazyeval#78)

Fixes #1400.

Fixes #1452.

Fixes #1719.

Fixes #2018.

Fixes #2125.

Fixes #2160.

Fixes #2223.

Fixes #2244.

[hadley]

For testing, that the hybrid evaluator is actually doing something it might be useful to implement #1691, and also define an in_hybrid() function that is defined as function() FALSE in R, and but returns TRUE in the C implementation.

[krlmlr]

Performance suffers, a faster Rcpp_eval() would help:

    

[codecov-io]

Current coverage is 64.32% (diff: 3.57%)

Merging #2190 into master will increase coverage by 0.38%

@@             master      #2190   diff @@
==========================================
  Files           194        193     -1   
  Lines          7967       7908    -59   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits           5094       5087     -7   
+ Misses         2873       2821    -52   
  Partials          0          0          

Powered by Codecov. Last update 93619cd...cb92df4

[krlmlr]

@hadley: Much better when we create + populate the environment with the active bindings only once (ad4f698). In the tested scenarios run time was slower by < 50%. We still can look into a faster Rcpp_eval(), but that would be orthogonal to this PR. I'll do some more local testing before suggesting a merge.

    

[hadley]

That doesn't look too bad at all.

(Also it looks like every big change is currently with both before and after commits. Probably on the the after label is necessary)

[krlmlr]

Thanks. I've thought about showing only "after", but I like having "before" as a way to double-check and resolve potential ambiguities. Please also note that for the level established at 2c095 in the "windowed" plot, no "before" entry is detected before 7d918.

[krlmlr]

@hadley: PTAL. I'll release bindrcpp to CRAN soon, but this shouldn't affect the review. Tests now show one warning, which should go with #2298.

NEWS entry:

• Breaking change: The column() and global() functions have been removed. They were never documented officially.
• Expressions in verbs are now interpreted correctly in many cases that failed before (e.g., use of $, case_when(), nonstandard evaluation, ...). This is possible, because these expressions are now evaluated in a specially constructed environment that retrieves column data on demand with the help of the bindrcpp package (Simplify hybrid evaluation #2190).

[hadley]

Could you do another benchmark plot please?

Should the news bullets also include mention of the new .env and .data pronouns? (And if you haven't implemented them yet, can you please do so?)

[krlmlr]

@hadley: Will do a benchmark plot. What should happen if the data frame contains columns named .env and/or .data?

[hadley]

@krlmlr I think our definition should win so if you wanted to access a column named .env you'd need to do .data$.env

[krlmlr]

@hadley: Done. Performance is very much comparable with master, see plots below. NEWS entries:

• Breaking change: The new .data and .env environments can be used inside all verbs that operate on data: .data$column_name accesses the column column_name, whereas .env$var accesses the external variable var. Columns or external variables named .data or .env are shadowed, use .data$... and/or .env$... to access them.
• Breaking change: The column() and global() functions have been removed. They were never documented officially. Use the new .data and .env environments instead.
• Expressions in verbs are now interpreted correctly in many cases that failed before (e.g., use of $, case_when(), nonstandard evaluation, ...). These expressions are now evaluated in a specially constructed environment that retrieves column data on demand with the help of the bindrcpp package (Simplify hybrid evaluation #2190).

Updated plots

[hadley]

The formatting of this seems a bit off

[hadley]

LGTM

[krlmlr]

@hadley: Added minor reformatting tweaks and NEWS, will squash-merge after CI has run. Thanks for the fast reviews.

[krlmlr]

@lionel-: This is the core of the new hybrid evaluator: Two new environments, one with the variables, one with .data and .env. The GroupedHybridCall class currently does too many things, splitting in #2315.

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/