Another attempt at an astable flag #298

pdeffebach · 2021-09-14T20:54:05Z

This attempt should be a more robust strategy than the current @t flag in DataFrameMacros.jl.

I will outline it in more detail shortly.

pdeffebach · 2021-09-15T14:17:26Z

More progress! Not ready for a review yet as i have not added a complete test set or begun documenting.

pdeffebach · 2021-09-16T14:03:46Z

Ready for a review!

This is a big PR. In the process, I have to finally remove some deprecated functionality for grouped data frames. In particular

@by df :a (n1 = :a, n2 = :b)

used to work. In the past release we had a warning that this needed to be

@by df :a $AsTable = (n1 = :a, n2 = :b)

and now we throw an error in the original expression.

Here is the docstring for @astable

`astable(args...)`

Return a NamedTuple from a transformation inside DataFramesMeta.jl macros.

@astable acts on a single block. It works through all top-level expressions
and collects all such expressions of the form :y = x, i.e. assignments to a
Symbol, which is a syntax error outside of the macro. At the end of the
expression, all assignments are collected into a NamedTuple to be used
with the AsTable destination in the DataFrames.jl transformation
mini-language.

Concretely, the expressions

df = DataFrame(a = 1)

@rtransform df @astable begin
    :x = 1
    y = 50
    :z = :x + y + :a
end

becomes the pair

function f(a)
    x_t = 1
    y = 50
    z_t = x_t + y + a

    (; x = x_t, z = z_t)
end

transform(df, [:a] => f => AsTable)

@astable is useful when performing intermediate calculations
yet store their results in new columns. For example, the following fails.

@rtransform df begin
    :new_col_1 = :x + :y
    :new_col_2 = :new_col_1 + :z
end

This because DataFrames.jl does not guarantee sequential evaluation of
transformations. @astable solves this problem

@rtransform df @astable begin
    :new_col_1 = :x + :y
    :new_col_2 = :new_col_1 + :z
end

Examples

julia> df = DataFrame(a = [1, 2, 3], b = [4, 5, 6]);

julia> d = @rtransform df @astable begin
           :x = 1
           y = 5
           :z = :x + y
       end
3×4 DataFrame
 Row │ a      b      x      z
     │ Int64  Int64  Int64  Int64
─────┼────────────────────────────
   1 │     1      4      1      6
   2 │     2      5      1      6
   3 │     3      6      1      6

julia> df = DataFrame(a = [1, 1, 2, 2], b = [5, 6, 70, 80]);

julia> @by df :a @astable begin
           $(DOLLAR)"Mean of b" = mean(:b)
           $(DOLLAR)"Standard deviation of b" = std(:b)
       end
2×3 DataFrame
 Row │ a      Mean of b  Standard deviation of b
     │ Int64  Float64    Float64
─────┼───────────────────────────────────────────
   1 │     1        5.5                 0.707107
   2 │     2       75.0                 7.07107

This implementation is more complicated than that of @t from DataFrameMacros.jl. In DataFrameMacros.jl, the following will fail

df = DataFrame(a = 1)
@transform df @t begin 
    :x = 1
    b + :x 
end

this is because it sends :x to src in src => fun => dest even though it doesn't exist in the DataFrame. @astable does not have this problem, at the cost of a more complicated implementation.

cc @bkamins @nalimilan

bkamins · 2021-09-16T14:09:19Z

transform(df, [:a] => f => AsTable)

I would think it should be transform(df, [:a] => ByRow(f) => AsTable)?

pdeffebach · 2021-09-16T14:17:32Z

Good catch. Will update.

nalimilan

Looks pretty cool! This will require a breaking release, right?

Using @astable to ensure operations are run sequentially is clever. The name is a bit surprising for this, but well... I also hope the compilation overhead isn't too large.

nalimilan · 2021-09-19T17:04:32Z

src/macros.jl

+julia> @by df :a @astable begin
+           $(DOLLAR)"Mean of b" = mean(:b)
+           $(DOLLAR)"Standard deviation of b" = std(:b)
+       end


This example can be achieved without @astable, right? Maybe do m = mean(:b); std(:b, mean=m) to illustrate the power of this function? Or, simpler, call extrema(:b) to create two columns.

Also, I wouldn't use long column names with spaces in them: better illustrate a single feature at a time.

great. changed.

src/parsing_astable.jl

src/parsing.jl

src/macros.jl

nalimilan · 2021-09-19T17:19:32Z

src/macros.jl

+`@astable` is useful when performing intermediate calculations
+yet store their results in new columns. For example, the following fails.
+
+```
+@rtransform df begin
+    :new_col_1 = :x + :y
+    :new_col_2 = :new_col_1 + :z
+end
+```
+
+This because DataFrames.jl does not guarantee sequential evaluation of
+transformations. `@astable` solves this problem


While this is an interesting side-effect, the main goal of AsTable is to allow returning multiple columns from a single "function". Probably worth mentioning? For example it's useful with extrema to compute the minimum and the maximum at the same time.

src/macros.jl

pdeffebach · 2021-09-21T13:27:11Z

Using @astable to ensure operations are run sequentially is clever. The name is a bit surprising for this, but well... I also hope the compilation overhead isn't too large.

This macro does more than that. It allows for local variables to per persistent as well. If I were to just force sequential transform calls, I would just create a transform call for each :y = f(:x) expression.

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

…eta.jl into astable_2

pdeffebach · 2021-09-21T15:54:14Z

This is ready for another round of reviews. I have added docs to the manual as well.

bkamins · 2021-09-21T17:19:31Z

docs/src/index.md

+information. 
+
+In a single block, all assignments of the form `:y = f(:x)` 
+or `$y = f(:x)` at the top-level are generate new columns.


Suggested change

or `$y = f(:x)` at the top-level are generate new columns.

or `$y = f(:x)` at the top-level generate new columns.

maybe add what $y has to resolve to (I understand it has to be Symbol, or strings are also accepted?)

Good catch. Turns out I was allowing unexpected behavior and patched the code.

src/macros.jl

bkamins · 2021-09-21T17:37:33Z

src/macros.jl

+
+Column assignment in `@astable` follows the same rules as
+column assignment more generally. Construct a new column
+from a string by escaping it with `$DOLLAR`.


can you add an example of this?

src/macros.jl

pdeffebach · 2021-09-22T22:14:13Z

Thanks, I've responded to the most recent round of reviews.

I have added more tests. I can't think of any new tests to add at the moment, they seem pretty well covered.

pdeffebach · 2021-09-22T22:17:47Z

cc @jkrumbiegel, if you want to review.

nalimilan · 2021-09-23T07:50:39Z

This macro does more than that. It allows for local variables to per persistent as well. If I were to just force sequential transform calls, I would just create a transform call for each :y = f(:x) expression.

Yes I know. What I'm saying is that mentioning sequential operations as the main justification for it was a bit weird.

src/macros.jl

src/parsing_astable.jl

nalimilan · 2021-09-23T07:44:04Z

test/astable_flag.jl

+        :b_max = ex[2]
+    end
+
+    @test sort(d.b_min) == [5, 7]


This kind of test is quite fragile. Better sort the whole data frame and compare it to the reference to make sure that groups and values match.

…eta.jl into astable_2

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

pdeffebach · 2021-09-23T14:48:15Z

I don't understand the changes to the github review interface, but I think i've addressed lingering issues.

I just added some checks to make sure you can't do @passmissing and @astable at the same time. I think this is intuitive in the @byrow case, but I want to finish #276 , which makes @passmissing work on column-wise transformations, before I do this.

pdeffebach · 2021-09-23T14:48:33Z

Tests pass and this can be merged.

bkamins · 2021-09-23T17:58:38Z

test/astable_flag.jl

+end
+
+@testset "@astable with just assignments, mutating" begin
+    # After finalizing above testset


@pdeffebach - this seems to be WIP?

Sorry about that. Thank you working on it now.

bkamins · 2021-09-23T17:59:32Z

test/astable_flag.jl

+
+    d = @rtransform df @astable begin
+        :x = 1
+


newline is not needed. Also I am not clear why you add nothing here and below? Does it change anything?

oh, I remember.

@transform df begin :x = 1 end

is valid code and does the same thing as the same with @astable does. So I wanted to test something that made sure it was hittng the @astable path and not the vanilla path.

I have deleted the extra new lines.

test/astable_flag.jl

pdeffebach

Thanks for the round of reviews. I think I have addressed everything. Sorry about forgetting some of the tests.

src/macros.jl

pdeffebach · 2021-09-23T18:04:39Z

test/astable_flag.jl

+end
+
+@testset "@astable with just assignments, mutating" begin
+    # After finalizing above testset


Sorry about that. Thank you working on it now.

pdeffebach · 2021-09-23T20:53:55Z

test/astable_flag.jl

+
+    d = @rtransform df @astable begin
+        :x = 1
+


oh, I remember.

@transform df begin :x = 1 end

is valid code and does the same thing as the same with @astable does. So I wanted to test something that made sure it was hittng the @astable path and not the vanilla path.

I have deleted the extra new lines.

test/astable_flag.jl

bkamins

Looks good. Just please check why Documenter fails before merging. Thank you!

docs/src/index.md

src/macros.jl

nalimilan · 2021-09-24T07:44:30Z

src/parsing.jl

+    println(MacroTools.prettify(fun))
+


Suggested change

println(MacroTools.prettify(fun))

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

pdeffebach · 2021-09-24T13:32:42Z

Thanks for the review. @nalimilan ready to merge?

bkamins · 2021-09-24T22:00:14Z

Bravo!

pdeffebach added 3 commits September 14, 2021 12:52

initial attempt

a8701c8

finally working

9b997a6

start adding tests

d639560

pdeffebach mentioned this pull request Sep 15, 2021

add astable macro #291

Closed

pdeffebach added 5 commits September 16, 2021 04:58

more tests

b77e8ca

more tests

3cdf0d5

add docstring

b878fbb

tests pass

2344a2e

add ByRow in docstring

6557def

nalimilan reviewed Sep 19, 2021

View reviewed changes

pdeffebach added 4 commits September 21, 2021 05:33

add type annotation

6002def

better docs

08a1c4b

more docs fixes

581b2cf

update index.md

7cc8947

pdeffebach and others added 2 commits September 21, 2021 09:38

Apply suggestions from code review

0eca67d

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

Merge branch 'astable_2' of https://github.com/pdeffebach/DataFramesM…

a4ab9a6

…eta.jl into astable_2

bkamins reviewed Sep 21, 2021

View reviewed changes

src/macros.jl Outdated Show resolved Hide resolved

bkamins reviewed Sep 21, 2021

View reviewed changes

src/macros.jl Show resolved Hide resolved

bkamins reviewed Sep 21, 2021

View reviewed changes

src/macros.jl Show resolved Hide resolved

bkamins reviewed Sep 21, 2021

View reviewed changes

pdeffebach added 3 commits September 22, 2021 06:29

clean named tuple creation

ab9bae4

add example with string

495f08a

grouping tests

01cb5e7

bkamins reviewed Sep 22, 2021

View reviewed changes

src/macros.jl Outdated Show resolved Hide resolved

bkamins reviewed Sep 22, 2021

View reviewed changes

src/macros.jl Show resolved Hide resolved

nalimilan reviewed Sep 23, 2021

View reviewed changes

pdeffebach and others added 8 commits September 23, 2021 06:08

changes

915191c

Merge branch 'astable_2' of https://github.com/pdeffebach/DataFramesM…

a331fc2

…eta.jl into astable_2

fix some errors

2ce4d9e

add macro check

57b4051

add errors for bad flag combo

da7674d

better grouping tests

285e3ac

Update src/parsing_astable.jl

713eaf0

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

add snipper to transform, select, combine, by

4e01c4a

add mutating tests

09c692a

bkamins reviewed Sep 23, 2021

View reviewed changes

test/astable_flag.jl Show resolved Hide resolved

pdeffebach commented Sep 23, 2021

View reviewed changes

bkamins approved these changes Sep 23, 2021

View reviewed changes

nalimilan reviewed Sep 24, 2021

View reviewed changes

pdeffebach and others added 2 commits September 24, 2021 05:03

get rid of debugging printin

ae26da8

Apply suggestions from code review

a7fd1a2

Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

nalimilan approved these changes Sep 24, 2021

View reviewed changes

pdeffebach merged commit cc066df into JuliaData:master Sep 24, 2021

pdeffebach deleted the astable_2 branch September 24, 2021 21:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Another attempt at an astable flag #298

Another attempt at an astable flag #298

pdeffebach commented Sep 14, 2021

pdeffebach commented Sep 15, 2021

pdeffebach commented Sep 16, 2021 •

edited by nalimilan

bkamins commented Sep 16, 2021

pdeffebach commented Sep 16, 2021

nalimilan left a comment

nalimilan Sep 19, 2021

pdeffebach Sep 21, 2021

nalimilan Sep 19, 2021

pdeffebach commented Sep 21, 2021

pdeffebach commented Sep 21, 2021

bkamins Sep 21, 2021

bkamins Sep 21, 2021

pdeffebach Sep 22, 2021

bkamins Sep 21, 2021

pdeffebach Sep 22, 2021

pdeffebach commented Sep 22, 2021

pdeffebach commented Sep 22, 2021

nalimilan commented Sep 23, 2021

nalimilan Sep 23, 2021

pdeffebach commented Sep 23, 2021 •

edited

pdeffebach commented Sep 23, 2021

bkamins Sep 23, 2021

pdeffebach Sep 23, 2021

bkamins Sep 23, 2021

pdeffebach Sep 23, 2021

pdeffebach left a comment

pdeffebach Sep 23, 2021

pdeffebach Sep 23, 2021

bkamins left a comment

nalimilan Sep 24, 2021

pdeffebach commented Sep 24, 2021

bkamins commented Sep 24, 2021

	or `$y = f(:x)` at the top-level are generate new columns.
	or `$y = f(:x)` at the top-level generate new columns.

Another attempt at an astable flag #298

Another attempt at an astable flag #298

Conversation

pdeffebach commented Sep 14, 2021

pdeffebach commented Sep 15, 2021

pdeffebach commented Sep 16, 2021 • edited by nalimilan

astable(args...)

Examples

bkamins commented Sep 16, 2021

pdeffebach commented Sep 16, 2021

nalimilan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeffebach commented Sep 21, 2021

pdeffebach commented Sep 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeffebach commented Sep 22, 2021

pdeffebach commented Sep 22, 2021

nalimilan commented Sep 23, 2021

Choose a reason for hiding this comment

pdeffebach commented Sep 23, 2021 • edited

pdeffebach commented Sep 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeffebach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkamins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeffebach commented Sep 24, 2021

bkamins commented Sep 24, 2021

pdeffebach commented Sep 16, 2021 •

edited by nalimilan

`astable(args...)`

pdeffebach commented Sep 23, 2021 •

edited