add more tests for window::shift and handle boundary cases #386

jimexist · 2021-05-31T02:14:01Z

Which issue does this PR close?

Closes #387.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

codecov-commenter · 2021-05-31T02:29:03Z

Codecov Report

Merging #386 (daf1655) into master (7ec3158) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #386      +/-   ##
==========================================
+ Coverage   82.62%   82.64%   +0.01%     
==========================================
  Files         162      162              
  Lines       44275    44310      +35     
==========================================
+ Hits        36582    36618      +36     
+ Misses       7693     7692       -1

Impacted Files	Coverage Δ
arrow/src/compute/kernels/window.rs	`100.00% <100.00%> (ø)`
parquet/src/encodings/encoding.rs	`95.04% <0.00%> (+0.19%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ec3158...daf1655. Read the comment docs.

nevi-me · 2021-06-02T01:06:48Z

arrow/src/compute/kernels/window.rs

-    let nulls = abs(offset as i64) as usize;
+    let value_len = values.len() as i64;
+    if offset == 0 {
+        Ok(values.slice(0, values.len()))


Same comment as #388

i guess it's not that easy - there's no easy way to clone array and likely suboptimal than just taking a slice here

nevi-me

One minor comment, LGTM otherwise

alamb · 2021-06-03T21:52:19Z

arrow/src/compute/kernels/window.rs

+/// let expected: Int32Array = vec![None, Some(4), None].into();
+/// assert_eq!(res.as_ref(), &expected);
+///
+/// // shift array 0 element, although not recommended


alamb

this one looked good to me. Epic tests @jimexist

alamb · 2021-06-04T19:00:24Z

FWIW I could not cherry-pick this automatically to active_release (what will become release version 4.3):

@ip-10-0-0-124:~/Software/arrow-rs$ CHERRY_PICK_SHA=45acc62    ARROW_GITHUB_API_TOKEN=$ARROW_GITHUB_API_TOKEN CHECKOUT_ROOT=/tmp/arrow-rs  python3 dev/release/cherry-pick-pr.py
CHERRY_PICK_SHA=45acc62    ARROW_GITHUB_API_TOKEN=$ARROW_GITHUB_API_TOKEN CHECKOUT_ROOT=/tmp/arrow-rs  python3 dev/release/cherry-pick-pr.py
Using checkout in /tmp/arrow-rs
Creating cherry pick from 45acc62c to cherry_pick_45acc62c
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0        
From github.com:apache/arrow-rs
 * branch              master     -> FETCH_HEAD
From github.com:apache/arrow-rs
 * branch              active_release -> FETCH_HEAD
Switched to a new branch 'cherry_pick_45acc62c'
error: could not apply 45acc62c... add more tests for window::shift and handle boundary cases (#386)
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
Command failed: ['git', 'cherry-pick', '45acc62']
With output:
--------------
b'Auto-merging arrow/src/compute/kernels/window.rs\nCONFLICT (content): Merge conflict in arrow/src/compute/kernels/window.rs\n'
--------------
Traceback (most recent call last):
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 153, in <module>
    make_cherry_pick()
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 116, in make_cherry_pick
    run_cmd(['git', 'cherry-pick', new_sha])
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 78, in run_cmd
    raise e
  File "/Users/alamb/Software/arrow-rs/dev/release/cherry-pick-pr.py", line 70, in run_cmd
    output = subprocess.check_output(cmd)
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'cherry-pick', '45acc62']' returned non-zero exit status 1.

If we want to include it we will have to backport it manually

* add more doc test for window::shift * handle i64::MIN first * use Ok(make_array(array.data_ref().clone()))

alamb · 2021-06-08T21:22:38Z

Second chance back porting has worked #429 👍

* add more doc test for window::shift * handle i64::MIN first * use Ok(make_array(array.data_ref().clone())) Co-authored-by: Jiayu Liu <Jimexist@users.noreply.github.com>

jimexist force-pushed the shift-edge-cases branch from efa5781 to a5a70ff Compare May 31, 2021 02:38

jimexist changed the title ~~add more doc test for window::shift~~ add more tests for window::shift and handle boundary cases May 31, 2021

jimexist mentioned this pull request May 31, 2021

implement lead and lag built-in window function apache/datafusion#429

Merged

nevi-me requested a review from Dandandan June 2, 2021 01:06

nevi-me reviewed Jun 2, 2021

View reviewed changes

nevi-me approved these changes Jun 2, 2021

View reviewed changes

Jiayu Liu added 3 commits June 3, 2021 08:31

add more doc test for window::shift

e63440f

handle i64::MIN first

20c4a74

use Ok(make_array(array.data_ref().clone()))

daf1655

jimexist force-pushed the shift-edge-cases branch from 64defcc to daf1655 Compare June 3, 2021 00:33

alamb reviewed Jun 3, 2021

View reviewed changes

alamb approved these changes Jun 3, 2021

View reviewed changes

alamb merged commit 45acc62 into apache:master Jun 3, 2021

alamb pushed a commit that referenced this pull request Jun 8, 2021

add more tests for window::shift and handle boundary cases (#386)

08fd10b

* add more doc test for window::shift * handle i64::MIN first * use Ok(make_array(array.data_ref().clone()))

alamb added the cherry-picked label Jun 8, 2021

alamb mentioned this pull request Jun 8, 2021

Cherry pick add more tests for window::shift and handle boundary cases to active_release #429

Merged

jimexist deleted the shift-edge-cases branch June 10, 2021 02:02

alamb mentioned this pull request Jun 10, 2021

Add changelog and bump version for proposed 4.3.0 release #444

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add more tests for window::shift and handle boundary cases #386

add more tests for window::shift and handle boundary cases #386

jimexist commented May 31, 2021 •

edited

Loading

codecov-commenter commented May 31, 2021 •

edited

Loading

nevi-me Jun 2, 2021

jimexist Jun 2, 2021

nevi-me left a comment

alamb Jun 3, 2021

alamb left a comment

alamb commented Jun 4, 2021

alamb commented Jun 8, 2021

add more tests for window::shift and handle boundary cases #386

add more tests for window::shift and handle boundary cases #386

Conversation

jimexist commented May 31, 2021 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

codecov-commenter commented May 31, 2021 • edited Loading

Codecov Report

nevi-me Jun 2, 2021

Choose a reason for hiding this comment

jimexist Jun 2, 2021

Choose a reason for hiding this comment

nevi-me left a comment

Choose a reason for hiding this comment

alamb Jun 3, 2021

Choose a reason for hiding this comment

alamb left a comment

Choose a reason for hiding this comment

alamb commented Jun 4, 2021

alamb commented Jun 8, 2021

jimexist commented May 31, 2021 •

edited

Loading

codecov-commenter commented May 31, 2021 •

edited

Loading