Make mutate use collecter h #2487

zeehio · 2017-03-02T00:57:52Z

This PR is a continuation to #2486 and closes #1892.

mutate(col2 = fun(col1)) on a grouped data frame calls fun once per group.

It used to require that fun returns the exact same type on each call. That is not desirable in functions that may return different (but compatible) types, such as integer and numeric.

This PR changes that behavior, so the returned vectors from each of the fun calls are combined using the same coercion rules than combine and bind_rows, defined in Collecter.h.

Comments are welcome. Feel free to be picky, so I can improve a bit my C++ and Rcpp skills. 😃

We want to support difftime in bind_rows and combine. We are already supporting mutate and I'm preparing a PR to make mutate use Collecter.h as well.

krlmlr

Thanks for looking into these issues! I have a few comments and questions.

krlmlr · 2017-03-02T08:21:50Z

tests/testthat/test-binds.R

@@ -457,3 +457,10 @@ test_that("bind_rows rejects data frame columns (#2015)", {
    fixed = TRUE
  )
 })
+
+test_that("bind_rows accepts difftime objects", {


Test case for "hms":

df1 <- data.frame(x = hms::hms(hours = 1)) df2 <- data.frame(x = as.difftime(1, units = "mins")) res <- bind_rows(df1, df2) expect_equal(res$x, hms::hms(hours = 1, minutes = 1))

krlmlr · 2017-03-02T08:23:48Z

inst/include/dplyr/Collecter.h

+    }
+
+    inline SEXP get() {
+      set_class(Parent::data, "difftime");


There is also the "hms" class that inherits from "difftime", I guess we'd like to return "hms" objects if the first object is of that class.

krlmlr · 2017-03-02T08:25:14Z

inst/include/dplyr/Collecter.h

+          double factor_data = time_conversion_factor(units);
+          if (factor_data != 1.0) {
+            for (int i=0; i<Parent::data.size(); i++) {
+              Parent::data[i] = factor_data*Parent::data[i];


Do we need to copy the data before in-place modification?

So, if a copy has been made, I'm missing it ;-)

krlmlr · 2017-03-02T08:26:55Z

inst/include/dplyr/Collecter.h

+          units = wrap("secs");
+          double factor_v = time_conversion_factor(v_units);
+          NumericVector v_sec(v);
+          double* v_sec_ptr = v_sec.begin();


This looks unsafe, better use an iterator and advance that (in addition to i).

krlmlr · 2017-03-02T08:27:36Z

inst/include/dplyr/Collecter.h

+        // then collect the data:
+        Parent::collect(index, v);
+      } else {
+        // We already units, is the new vector with the same units?


Please check grammar.

krlmlr · 2017-03-02T08:29:01Z

inst/include/dplyr/Collecter.h

+      }
+      if (units.isNULL()) {
+        // if current unit is NULL, grab the new one
+        units = v_units;


Can we simply reject "difftime" objects that don't have a "units" attribute?

krlmlr · 2017-03-02T08:31:18Z

inst/include/dplyr/Collecter.h

@@ -25,12 +25,13 @@ namespace dplyr {
  static bool is_class_known(SEXP x) {
    /* C++11 (need initializer lists)
    static std::set<std::string> known_classes {
-      "POSIXct", "factor", "Date", "AsIs", "integer64", "table"
+      "difftime", "POSIXct", "factor", "Date", "AsIs", "integer64", "table"


We shouldn't need to carry around code in comments. The old-style code also works in C++11, we'll rewrite that when we are on C++11 and when we touch the code again.

krlmlr · 2017-03-02T08:36:13Z

inst/include/dplyr/Gatherer.h

-      if (first_non_na < gdf.ngroups())
-        grab(first, indices);
-      copy_most_attributes(data, first);
+class Gatherer {


Could you please indent like the original, to make review simpler?

krlmlr · 2017-03-02T08:37:31Z

@zeehio: If you address the feedback here and backport to #2486, we can continue the discussion there.

`mutate(col2 = fun(col1))` on a grouped data frame calls `fun` once per group. It used to require that `fun` returns the exact same type and that was not desirable in functions that may return different (but compatible) types, such as integer and numeric. This PR changes that behaviour, so the returned vectors from each of the `fun` calls are combined using the same coercion rules than `combine` and `bind_rows`, defined in `Collecter.h`.

zeehio · 2017-03-02T22:28:26Z

@krlmlr I have taken care of all the issues you mentioned. Apologies for the grammar error and for all the indentation issues, thanks for all the comments and suggestions 👍

krlmlr · 2017-03-02T22:48:17Z

inst/include/dplyr/Collecter.h

+          double factor_data = time_conversion_factor(units);
+          if (factor_data != 1.0) {
+            for (int i=0; i<Parent::data.size(); i++) {
+              Parent::data[i] = factor_data*Parent::data[i];


So, if a copy has been made, I'm missing it ;-)

krlmlr · 2017-03-02T22:50:57Z

inst/include/dplyr/Collecter.h

+            for (int i=0; i<index.size(); i++) {
+              Parent::data[index[i]] = factor_v * REAL(v)[i];
+            }
+          } else if (TYPEOF(v) == INTSXP) {


How do you create a difftime of mode integer?

I can only think of structure(4L, units="secs", class = "difftime") and I agree it is forcing it. I can drop the else if if you prefer.

krlmlr · 2017-03-02T22:53:37Z

inst/include/dplyr/Collecter.h

+    double time_conversion_factor(RObject v_units) {
+      // Acceptable units based on r-source/src/library/base/R/datetime.R
+      double factor = 1;
+      std::string v_units_c = Rcpp::as<std::string>(v_units);


I wounder if we could use a map (as a static variable in a function) that allows lookup by SEXP, both here and in has_valid_time_unit().

In the last commit I have used an std::string as key. If that is not good enough I will try to fix it tomorrow.

krlmlr · 2017-03-02T22:53:52Z

inst/include/dplyr/Collecter.h

+    }
+
+  private:
+    RObject units;


Can you please move data members to the bottom?

hadley · 2017-03-03T00:34:00Z

inst/include/dplyr/Collecter.h

+      }
+    }
+
+    double time_conversion_factor(SEXP v_units) {


This feels overworked to me now - I preferred the previous version with explicit if statements. Might be better to make the argument std::string()

hadley · 2017-03-03T00:35:15Z

inst/include/dplyr/Collecter.h

+    }
+
+    void collect_difftime(const SlicingIndex& index, SEXP v) {
+      RObject v_units(Rf_getAttrib(v, Rf_install("units")));


Can't you do v.attr("units")? I'm pretty sure there's a C++ api here.

hadley · 2017-03-03T00:35:59Z

inst/include/dplyr/Collecter.h

+    void collect_difftime(const SlicingIndex& index, SEXP v) {
+      RObject v_units(Rf_getAttrib(v, Rf_install("units")));
+      if (v_units.isNULL()) {
+        stop("Can't collect difftime without units");


I think here, and for the non REALSXP case below, you can simply do stop("Invalid difftime object").

hadley · 2017-03-03T00:36:37Z

inst/include/dplyr/Collecter.h

+      }
+    }
+
+    RObject units;


Would all this code be simpler if units was a std::string?

zeehio · 2017-03-04T19:38:12Z

I took care of the comments, feel free to do another review if you want

hadley · 2017-03-05T08:24:31Z

Looks good, thanks!

lock · 2019-01-18T16:32:51Z

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/

zeehio added 2 commits March 2, 2017 01:14

Add difftime support to Collecter.h

a13f046

We want to support difftime in bind_rows and combine. We are already supporting mutate and I'm preparing a PR to make mutate use Collecter.h as well.

Fix oldrel test

8dec9e8

krlmlr reviewed Mar 2, 2017

View reviewed changes

zeehio force-pushed the make_mutate_use_collecter_h branch from 1f1b494 to 36319d8 Compare March 2, 2017 18:03

zeehio changed the title ~~Make mutate use collecter h~~ [WIP] Make mutate use collecter h Mar 2, 2017

zeehio force-pushed the make_mutate_use_collecter_h branch from 9b0fca5 to dad85f8 Compare March 2, 2017 19:49

Add hms support and other @krlmlr feedback

1231ced

zeehio force-pushed the make_mutate_use_collecter_h branch from dad85f8 to 1231ced Compare March 2, 2017 20:57

zeehio changed the title ~~[WIP] Make mutate use collecter h~~ Make mutate use collecter h Mar 2, 2017

krlmlr reviewed Mar 2, 2017

View reviewed changes

Further suggestions to PR tidyverse#2487 by @krlmlr

c920544

hadley reviewed Mar 3, 2017

View reviewed changes

Difftime collecter corrections suggested by @hadley

48f90e0

hadley merged commit 996318b into tidyverse:master Mar 5, 2017

zeehio deleted the make_mutate_use_collecter_h branch March 5, 2017 11:56

zeehio mentioned this pull request Feb 16, 2018

Fix #407. unnest works with S3 vectors. tidyverse/tidyr#419

Closed

lock bot locked and limited conversation to collaborators Jan 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make mutate use collecter h #2487

Make mutate use collecter h #2487

zeehio commented Mar 2, 2017

krlmlr left a comment

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr commented Mar 2, 2017

zeehio commented Mar 2, 2017

krlmlr Mar 2, 2017

krlmlr Mar 2, 2017

zeehio Mar 2, 2017

krlmlr Mar 2, 2017

zeehio Mar 3, 2017 •

edited

Loading

krlmlr Mar 2, 2017

hadley Mar 3, 2017

hadley Mar 3, 2017

hadley Mar 3, 2017

hadley Mar 3, 2017

zeehio commented Mar 4, 2017

hadley commented Mar 5, 2017

lock bot commented Jan 18, 2019

Make mutate use collecter h #2487

Make mutate use collecter h #2487

Conversation

zeehio commented Mar 2, 2017

krlmlr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krlmlr commented Mar 2, 2017

zeehio commented Mar 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeehio Mar 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeehio commented Mar 4, 2017

hadley commented Mar 5, 2017

lock bot commented Jan 18, 2019

zeehio Mar 3, 2017 •

edited

Loading