morph() to automatically remove columns "used up" by a mutate()

[ArtemSokolov]

Dear dplyr developers,

A recent Stack Overflow question raised an interesting use case of having columns fed to a mutate() call automatically removed from the result. To do this, the mutator would need to parse the input expressions to determine what symbols were used, and I made the first pass at designing such a function. The question author liked my answer and suggested that I contribute it to dplyr.

I am happy to work on a PR with a more robust implementation, but I wanted to check with you if such a feature would align with your design principles and the spirit of the package.

Thanks. Big fan of your work.
-Artem

[krlmlr]

library(tidyverse)

#iris %>% transmutate(Petal.Area = Petal.Width * Petal.Length)
iris %>%
  as_tibble() %>% 
  mutate(Petal.Area = Petal.Width * Petal.Length) %>% 
  select(-Petal.Width, -Petal.Length)
#> # A tibble: 150 x 4
#>    Sepal.Length Sepal.Width Species Petal.Area
#>           <dbl>       <dbl> <fct>        <dbl>
#>  1          5.1         3.5 setosa       0.280
#>  2          4.9         3   setosa       0.280
#>  3          4.7         3.2 setosa       0.26 
#>  4          4.6         3.1 setosa       0.3  
#>  5          5           3.6 setosa       0.280
#>  6          5.4         3.9 setosa       0.68 
#>  7          4.6         3.4 setosa       0.42 
#>  8          5           3.4 setosa       0.3  
#>  9          4.4         2.9 setosa       0.280
#> 10          4.9         3.1 setosa       0.15 
#> # ... with 140 more rows

Created on 2018-07-21 by the reprex package (v0.2.0).

Thanks, I'm missing this functionality myself occasionally. Instead of parsing the expression, we could detect and record column access in the C++ code.

What should the verb do in a grouped scenario, if some groups access a different set of columns than other groups?

[ArtemSokolov]

I think there are two natural options: union - remove all columns that are accessed by at least one group, or intersection - remove only the columns that are accessed by all groups. It might be nice to be able to specify which of the two should be used, but I'm not sure where such an option would go...

[krlmlr]

I'd rather support only intersection, but that might be more difficult to implement than union.

The biggest problem I see with both options is that type stability is compromised -- the resulting data frame might end up with different columns, depending on the data. Perhaps the safest thing to do would be to raise an error if different columns are accessed for different groups from this verb.

What naming alternatives do we have? It might be difficult to remember the differences between mutate(), transmute() and transmutate().

[ArtemSokolov]

I agree that perhaps the appropriate action for accessing different columns across groups is to raise an error. I'm not sure I follow the type stability concerns; if the intersection of all accessed columns is removed from each group, is that not a consistent transformation of each group?

For naming alternatives, perhaps we can turn to thesaurus: https://www.thesaurus.com/browse/transmute
If it doesn't become too annoying to type, metamorphose() might be a viable option.

EDIT: A nicer alternative might be alter(), as taken from https://www.thesaurus.com/browse/mutate

[krlmlr]

Suppose we have two group types: X and Y, the mutator code for group X accesses column a, for Y column b. If only groups of type X are present in the data, column a is accessed and removed; if both types are present, both columns a and b are accessed -- which to remove? Both intersect and union produce results inconsistent with the first scenario.

We create something, but also take something else away. How about trade() ?

[ArtemSokolov]

Thanks for the example, Kirill. That makes sense, and raising an error seems like the best approach to maintain consistency.

I think I have a slight preference towards alter(), because it is semantically similar to mutate() and transmute(). But trade() is a good choice as well!

[krlmlr]

morph() ?

[ArtemSokolov]

Yes!! morph() is perfect.

[vlepori]

Hi, just chipping in to say I would find this feature very useful too. I like the transmutate() name, as it is indeed a mixture of the two existing functions. Otherwise, I could see it being implemented as an additional argument to transmute(), as in drop.all = FALSE.

Thanks !

[dekaufman]

Hi! As the originator of the request on StackOverflow, great thanks to Artem for making it work and to all of you for picking up the banner.

Making this an optional argument to transmute() as Vasco suggests would perfectly represent my initial wish for the function simply to work a little differently on request. And then there's no difficulty in picking a new word.

If you make it a separate function, I like either morph() or transmutate().

[moodymudskipper]

I'd also like it better if it was integrated into transmute rather than adding a new verb (which would again be almost a synonym of merge, transform, gather... in English), with a unique additional parameter that would support the intersect and union options mentioned above.

It could also be a parameter of mutate where we'd have keep either "all" (default), "new" (default transmute behavior), "intersection" or "union", and then transmute would just be a shorthand for mutate with keep = "new".

[mkoohafkan]

tidygraph already uses morph for temporary graph topology modification, and it would be a shame to add this conflict.

I think that a whole new verb for a slight modification to mutate/transmute is overkill. An argument to transmute seems pretty reasonable, it just makes the decision of what to do with grouping variables explicit.

[krlmlr]

I like the idea of the .keep = "all" argument to mutate(), or perhaps .remove = "none" (with options "other", "used" and "used_once"). But it's rather low priority now.