Merge pull request #92 from TidierOrg/parse-blocks

NEWS.md

@@ -1,5 +1,9 @@
 # TidierData.jl updates
 
+## v0.15.0 - 2024-02-25
+- Add support for `begin-end` blocks for all macros accepting multiple expressions
+- Bug fix to add support for expressions inside of `@group_by()`, as in `@group_by(b = a + 1)`
+
 ## v0.14.7 - 2024-02-16
 - Bug fix to allow `PackageName.function()` within macros to be used without escaping
 

Project.toml

@@ -1,7 +1,7 @@
 name = "TidierData"
 uuid = "fe2206b3-d496-4ee9-a338-6a095c4ece80"
 authors = ["Karandeep Singh"]
-version = "0.14.7"
+version = "0.15.0"
 
 [deps]
 Chain = "8be319e6-bccf-4806-a6f7-6fae938471bc"

README.md

@@ -13,7 +13,7 @@ TidierData.jl is a 100% Julia implementation of the dplyr and tidyr R packages.
 extensive meta-programming capabilities, TidierData.jl is an R user’s love
 letter to data analysis in Julia.
 
-`TidierData.jl` has three goals, which differentiate it from other data analysis
+`TidierData.jl` has two goals, which differentiate it from other data analysis
 meta-packages in Julia:
 
 1.  **Stick as closely to dplyr and tidyr syntax as possible:** Whereas other
@@ -30,16 +30,6 @@ meta-packages in Julia:
     automatically vectorized. Read the documentation page on "Autovectorization"
     to read about how this works, and how to override the defaults.
 
-3.  **Make scalars and tuples mostly interchangeable:** In Julia, the function
-    `across(a, mean)` is dispatched differently than `across((a, b), mean)`.
-    The first argument in the first instance above is treated as a scalar,
-    whereas the second instance is treated as a tuple. This can be very confusing
-    to R users because `1 == c(1)` is `TRUE` in R, whereas in Julia `1 == (1,)`
-    evaluates to `false`. The design philosophy in `TidierData.jl` is that the user
-    should feel free to provide a scalar or a tuple as they see fit anytime
-    multiple values are considered valid for a given argument, such as in
-    `across()`, and `TidierData.jl` will figure out how to dispatch it.
-
 ## Installation
 
 For the stable version:

docs/examples/UserGuide/piping.jl

@@ -0,0 +1,131 @@
+# The easiest way to use TidierData.jl for complex data transformation operations is to connect them together using pipes. Julia comes with the built-in `|>` pipe operator, but TidierData.jl also includes and re-exports the `@chain` macro from the Chain.jl package. On this page, we will show you how to use both approaches.
+
+# First, let's load a dataset.
+
+using TidierData
+using RDatasets
+
+movies = dataset("ggplot2", "movies");
+
+# ## Julia's built-in `|>` pipe
+
+# If we wanted to figure out the number of rows in the `movies` data frame, one way to do this is to apply the `nrow()` function to movies. The most straightforward way is to write it like this:
+
+nrow(movies)
+
+# Another perfectly valid way to write this expression is by piping `movies` into `nrow` using the `|>` pipe operator.
+
+movies |> nrow
+
+# Why might we want to do this? Well, whereas the first expression would naturally be read as "Calculate the number of rows of movies," the second expression reads as "Start with movies, then calculate the number of rows." For a simple expression, these are easy enough to reason about. However, as we start to pipe more and more functions in a single expression, the piped version becomes much easier to reason about.
+
+# One quick note about Julia's built-in pipe: writing `movies |> nrow()` would *not* be considered valid. This is because Julia's built-in pipe always expects a function and *not* a function call. Writing `nrow` by itself is *naming* the function, whereas writing `nrow()` is *calling* the function. This quickly becomes an issue once we want to supply arguments to the function we are calling.
+
+# Consider another approach to calculating the number of rows:
+
+size(movies, 1)
+
+# In this case, the `size()` function returns a tuple of `(rows, columns)`, and if you supply an optional second argument specifying the index of the tuple, it returns only that dimension. In this case, we called `size()` with a second argument of `1`, indicating that we only wanted the function to return the number of rows.
+
+# How would we write this using Julia's built-in pipe?
+
+movies |> 
+  x -> size(x, 1)
+
+# You might have wanted to write `movies |> size(1)`, but because `size(1)` would represent a function *call*, we have to wrap the function call within an anonymous function, which is easily accomplished using the `x -> func(x, arg1, arg2)` syntax, where `func()` refers to any function and `arg1` and `arg2` refer to any additional arguments that are needed.
+
+# Another way we could have accomplished this is to calculate `size`, which returns a tuple of `(rows, columns)`, and then to use an anonymous function to grab the first value. Since we are calculating `size` without any arguments, we can simply write `size` within the pipe. However, to grab the first value using the `x[1]` syntax, we have to define an anonymous function. Putting it all together, we get this approach to piping:
+
+movies |>
+  size |>
+  x -> x[1]
+
+# ## Using the `@chain` macro
+
+# The `@chain` macro comes from the Chain.jl package and is included and re-exported by TidierData.jl. Let's do this same series of exercises using `@chain`.
+
+# Let's calculate the number of rows using `@chain`.
+
+@chain movies nrow
+
+# One of the reasons we prefer the use of `@chain` in TidierData.jl is that it is so concise. There is no need for any operator. Another interesting thing is that `@chain` doesn't care whether you use a function *name* or a function *call*. Both approaches work. As a result, writing `nrow()` instead of `nrow` is equally valid using `@chain`.
+
+@chain movies nrow()
+
+# There are two options for writing out multi-row chains. The preferred approach is as follows, where the starting item is listed, followed by a `begin-end` block.
+
+@chain movies begin
+  nrow
+end
+
+# `@chain` also comes with a built-in placeholder, which is `\_`. To calculate the `size` and extract the first value, we can use this approach:
+
+@chain movies begin
+  size
+  _[1]
+end
+
+# You don't have to list the data frame before the `begin-end` block. This is equally valid:
+
+@chain begin
+  movies
+  size
+  _[1]
+end
+
+# The only time this approach is preferred is when instead of simply naming the data frame, you are using a function to read in the data frame from a file or database. Because this function call may include the path of the file, which could be quite long, it's easier to write this on it's own line within the `begin-end` block.
+
+# While the documentation for TidierData.jl follows the convention of placing piped functions on separate lines of code using `begin-end` blocks, this is purely convention for ease of readability. You could rewrite the code above without the `begin-end` block as follows:
+
+@chain movies size _[1]
+
+# For simple transformations, this approach is both concise and readable.
+
+# ## Using `@chain` with TidierData.jl
+
+# Returning to our convention of multi-line pipes, let's grab the first five movies that were released since 2000 and had a rating of at least 9 out of 10. Here is one way that we could write this:
+
+@chain movies begin
+    @filter(Year >= 2000 && Rating >= 9)
+    @slice(1:5)
+end
+
+# Note: we generally prefer using `&&` in Julia because it is a "short-cut" operator. If the first condition evaluates to `false`, then the second condition is not even evaluated, which makes it faster (because it takes a short-cut).
+
+# In the case of `@filter`, multiple conditions can be written out as separate expressions.
+
+@chain movies begin
+  @filter(Year >= 2000, Rating >= 9)
+  @slice(1:5)
+end
+
+# Another to write this expression is take advantage of the fact that Julia macros can be called without parentheses. In this case, we will add back the `&&` for the sake of readability.
+
+@chain movies begin
+  @filter Year >= 2000 && Rating >= 9
+  @slice 1:5
+end
+
+# Lastly, TidierData.jl also supports multi-line expressions within each of the macros that accept multiple expressions. So you could also write this as follows:
+
+@chain movies begin
+  @filter begin
+    Year >= 2000
+    Rating >= 9
+  end
+  @slice 1:5
+end
+
+# What's nice about this approach is that if you want to remove some criteria, you can easily comment out the relevant parts. For example, if you're willing to consider older movies, just comment out the `Year >= 2000`.
+
+@chain movies begin
+  @filter begin
+    ## Year >= 2000
+    Rating >= 9
+  end
+  @slice 1:5
+end
+
+# ## Which approach to use?
+
+# The purpose of this page was to show you that both Julia's native pipes and the `@chain` macro are perfectly valid and capable. We prefer the use of `@chain` because it is a bit more flexible and concise, with a syntax that makes it easy to comment out individual operations. We have adopted a similar `begin-end` block functionality within TidierData.jl itself, so that you can spread arguments out over multiple lines if you prefer. In the end, the choice is up to you!

docs/mkdocs.yml

@@ -127,6 +127,7 @@ nav:
   - "@arrange" : "examples/generated/UserGuide/arrange.md"
   - "@distinct" : "examples/generated/UserGuide/distinct.md"
   - "across" : "examples/generated/UserGuide/across.md"
+  - "Piping" : "examples/generated/UserGuide/piping.md"
   - "Conditionals": "examples/generated/UserGuide/conditionals.md"
   - "Joins" : "examples/generated/UserGuide/joins.md"
   - "Binding" : "examples/generated/UserGuide/binding.md" 

docs/src/index.md

@@ -8,7 +8,7 @@ Powered by the DataFrames.jl package and Julia’s
 extensive meta-programming capabilities, TidierData.jl is an R user’s love
 letter to data analysis in Julia.
 
-`TidierData.jl` has three goals, which differentiate it from other data analysis
+`TidierData.jl` has two goals, which differentiate it from other data analysis
 meta-packages in Julia:
 
 ```@raw html
@@ -45,11 +45,6 @@ meta-packages in Julia:
     Broadcasting trips up many R users switching to Julia because R users are used to most functions being vectorized. `TidierData.jl` currently uses a lookup table to decide which functions *not* to vectorize; all other functions are automatically vectorized. Read the documentation page on "Autovectorization" to read about how this works, and how to override the defaults. An example of where this issue commonly causes errors is when centering a variable. To create a new column `a` that centers the column `b`, `TidierData.jl` lets you simply write `a = b - mean(b)` exactly as you would in R. This works because `TidierData.jl` knows to *not* vectorize `mean()` while also recognizing that `-` *should* be vectorized such that this expression is rewritten in `DataFrames.jl` as `:b => (b -> b .- mean(b)) => :a`. For any user-defined function that you want to "mark" as being non-vectorized, you can prefix it with a `~`. For example, a function `new_mean()`, if it had the same functionality as `mean()` *would* normally get vectorized by `TidierData.jl` unless you write it as `~new_mean()`.
 ```
 
-```@raw html
-??? tip "Make scalars and tuples mostly interchangeable."
-    In Julia, the function `across(a, mean)` is dispatched differently than `across((a, b), mean)`. The first argument in the first instance above is treated as a scalar, whereas the second instance is treated as a tuple. This can be very confusing to R users because `1 == c(1)` is `TRUE` in R, whereas in Julia `1 == (1,)` evaluates to `false`. The design philosophy in `TidierData.jl` is that the user should feel free to provide a scalar or a tuple as they see fit anytime multiple values are considered valid for a given argument, such as in `across()`, and `TidierData.jl` will figure out how to dispatch it.
-```
-
 ## Installation
 
 For the stable version:

src/TidierData.jl

@@ -73,6 +73,7 @@ end
 $docstring_select
 """
 macro select(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -131,6 +132,7 @@ end
 $docstring_transmute
 """
 macro transmute(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -189,6 +191,7 @@ end
 $docstring_rename
 """
 macro rename(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -247,6 +250,7 @@ end
 $docstring_mutate
 """
 macro mutate(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -305,6 +309,7 @@ end
 $docstring_summarize
 """
 macro summarize(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs; from_summarize = true)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -376,6 +381,7 @@ end
 $docstring_filter
 """
 macro filter(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -434,6 +440,7 @@ end
 $docstring_group_by
 """
 macro group_by(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -444,7 +451,7 @@ macro group_by(df, exprs...)
   grouping_exprs = parse_group_by.(exprs)
 
   df_expr = quote
-    local any_expressions = all(typeof.($tidy_exprs) .!= QuoteNode)
+    local any_expressions = any(typeof.($tidy_exprs) .!= QuoteNode)
 
     if $any_found_n || $any_found_row_number || any_expressions
       if $(esc(df)) isa GroupedDataFrame
@@ -494,6 +501,7 @@ end
 $docstring_arrange
 """
 macro arrange(df, exprs...)
+  exprs = parse_blocks(exprs...)
   arrange_exprs = parse_desc.(exprs)
   df_expr = quote
     if $(esc(df)) isa GroupedDataFrame
@@ -518,6 +526,7 @@ end
 $docstring_distinct
 """
 macro distinct(df, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]
@@ -620,6 +629,7 @@ end
 $docstring_rename_with
 """
 macro rename_with(df, fn, exprs...)
+  exprs = parse_blocks(exprs...)
   interpolated_exprs = parse_interpolation.(exprs)
 
   tidy_exprs = [i[1] for i in interpolated_exprs]

src/binding.jl

@@ -2,6 +2,7 @@
 $docstring_bind_rows
 """
 macro bind_rows(df, exprs...)
+  exprs = parse_blocks(exprs...)
   tidy_exprs = parse_bind_args.(exprs)
   locate_id = findfirst(i -> i[2], tidy_exprs)
   if locate_id isa Nothing
@@ -23,6 +24,7 @@ end
 $docstring_bind_cols
 """
 macro bind_cols(df, exprs...)
+  exprs = parse_blocks(exprs...)
   tidy_exprs = parse_bind_args.(exprs)
   df_vec = [i[1] for i in tidy_exprs]
 

src/compound_verbs.jl

@@ -6,6 +6,7 @@
 $docstring_tally
 """
 macro tally(df, exprs...)
+  exprs = parse_blocks(exprs...)
   wt, sort = parse_tally_args(exprs...)
 
   wt_quoted = QuoteNode(wt)
@@ -51,6 +52,7 @@ end
 $docstring_count
 """
 macro count(df, exprs...)
+  exprs = parse_blocks(exprs...)
   col_names, wt, sort = parse_count_args(exprs...)
 
   col_names_quoted = QuoteNode(col_names)

src/docstrings.jl

@@ -424,7 +424,24 @@ rows as `df`.
 julia> df = DataFrame(a = 'a':'e', b = 1:5, c = 11:15);
 
 julia> @chain df begin
-         @mutate(d = b + c, b_minus_mean_b = b - mean(b))
+         @mutate(d = b + c,
+                 b_minus_mean_b = b - mean(b))
+       end
+5×5 DataFrame
+ Row │ a     b      c      d      b_minus_mean_b 
+     │ Char  Int64  Int64  Int64  Float64        
+─────┼───────────────────────────────────────────
+   1 │ a         1     11     12            -2.0
+   2 │ b         2     12     14            -1.0
+   3 │ c         3     13     16             0.0
+   4 │ d         4     14     18             1.0
+   5 │ e         5     15     20             2.0
+
+julia> @chain df begin
+         @mutate begin
+           d = b + c
+           b_minus_mean_b = b - mean(b)
+         end
        end
 5×5 DataFrame
  Row │ a     b      c      d      b_minus_mean_b 
@@ -511,14 +528,27 @@ Create a new DataFrame with one row that aggregating all observations from the i
 julia> df = DataFrame(a = 'a':'e', b = 1:5, c = 11:15);
 
 julia> @chain df begin
-         @summarize(mean_b = mean(b), median_b = median(b))
+         @summarize(mean_b = mean(b),
+                    median_b = median(b))
        end
 1×2 DataFrame
  Row │ mean_b   median_b 
      │ Float64  Float64  
 ─────┼───────────────────
    1 │     3.0       3.0
-  
+
+julia> @chain df begin
+         @summarize begin
+           mean_b = mean(b)
+           median_b = median(b)
+         end
+       end
+1×2 DataFrame
+ Row │ mean_b   median_b 
+     │ Float64  Float64  
+─────┼───────────────────
+   1 │     3.0       3.0 
+
 julia> @chain df begin
          @summarise(mean_b = mean(b), median_b = median(b))
        end