[SPARK-7836] [SPARK-7822] Python API of window functions #6364

davies · 2015-05-22T22:02:49Z

This PR also fix ntile() to accept only integer.

rxin · 2015-05-22T22:08:44Z

python/pyspark/sql/functions.py

+@since(1.4)
+def ntile(n):
+    """
+    NTILE for specified column or expression.


add "Window function:" ?

I will submit a patch to udpate scaladoc for all scala methods. the docs are not there yet.

SparkQA · 2015-05-23T00:11:39Z

Test build #33367 has finished for PR 6364 at commit 778e2c0.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Window(object):
- class WindowSpec(object):

rxin · 2015-05-23T00:14:04Z

Looks like it failed in Python 3?

SparkQA · 2015-05-23T00:21:12Z

Test build #33369 has finished for PR 6364 at commit 2649358.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Window(object):
- class WindowSpec(object):

rxin · 2015-05-23T01:15:43Z

python/pyspark/sql/functions.py

@@ -130,12 +141,25 @@ def _(col1, col2):
    'pow': 'Returns the value of the first argument raised to the power of the second argument.'
 }

+_window_functions = {
+    'lag': """returns the lag value of current row of the expression,


can we update the documentation here?

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/functions.scala

davies · 2015-05-23T01:36:31Z

@rxin Had updated

SparkQA · 2015-05-23T03:38:11Z

Test build #33385 has finished for PR 6364 at commit 62bbb6c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class Window(object):
- class WindowSpec(object):

SparkQA · 2015-05-23T03:59:37Z

Test build #33386 has finished for PR 6364 at commit 66092b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "public class " + className + extendsText + " implements java.io.Serializable
- class Window(object):
- class WindowSpec(object):

SparkQA · 2015-05-23T04:02:05Z

Test build #853 has finished for PR 6364 at commit 62bbb6c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-05-23T05:41:25Z

I think this still needs sign off from @rxin

[SPARK-7836] [SPARK-7822] Python API of window functions

davies · 2015-05-23T05:54:47Z

Yes, @rxin is working on it.

rxin · 2015-05-23T05:58:13Z

I submitted a pull request with more documentation changes: #6374

…elated updates 1. ntile should take an integer as parameter. 2. Added Python API (based on #6364) 3. Update documentation of various DataFrame Python functions. Author: Davies Liu <davies@databricks.com> Author: Reynold Xin <rxin@databricks.com> Closes #6374 from rxin/window-final and squashes the following commits: 69004c7 [Reynold Xin] Style fix. 288cea9 [Reynold Xin] Update documentaiton. 7cb8985 [Reynold Xin] Merge pull request #6364 from davies/window 66092b4 [Davies Liu] update docs ed73cb4 [Reynold Xin] [SPARK-7322][SQL] Improve DataFrame window function documentation. ef55132 [Davies Liu] Merge branch 'master' of github.com:apache/spark into window4 8936ade [Davies Liu] fix maxint in python 3 2649358 [Davies Liu] update docs 778e2c0 [Davies Liu] SPARK-7836 and SPARK-7822: Python API of window functions (cherry picked from commit efe3bfd) Signed-off-by: Yin Huai <yhuai@databricks.com>

…elated updates 1. ntile should take an integer as parameter. 2. Added Python API (based on #6364) 3. Update documentation of various DataFrame Python functions. Author: Davies Liu <davies@databricks.com> Author: Reynold Xin <rxin@databricks.com> Closes #6374 from rxin/window-final and squashes the following commits: 69004c7 [Reynold Xin] Style fix. 288cea9 [Reynold Xin] Update documentaiton. 7cb8985 [Reynold Xin] Merge pull request #6364 from davies/window 66092b4 [Davies Liu] update docs ed73cb4 [Reynold Xin] [SPARK-7322][SQL] Improve DataFrame window function documentation. ef55132 [Davies Liu] Merge branch 'master' of github.com:apache/spark into window4 8936ade [Davies Liu] fix maxint in python 3 2649358 [Davies Liu] update docs 778e2c0 [Davies Liu] SPARK-7836 and SPARK-7822: Python API of window functions

…elated updates 1. ntile should take an integer as parameter. 2. Added Python API (based on apache#6364) 3. Update documentation of various DataFrame Python functions. Author: Davies Liu <davies@databricks.com> Author: Reynold Xin <rxin@databricks.com> Closes apache#6374 from rxin/window-final and squashes the following commits: 69004c7 [Reynold Xin] Style fix. 288cea9 [Reynold Xin] Update documentaiton. 7cb8985 [Reynold Xin] Merge pull request apache#6364 from davies/window 66092b4 [Davies Liu] update docs ed73cb4 [Reynold Xin] [SPARK-7322][SQL] Improve DataFrame window function documentation. ef55132 [Davies Liu] Merge branch 'master' of github.com:apache/spark into window4 8936ade [Davies Liu] fix maxint in python 3 2649358 [Davies Liu] update docs 778e2c0 [Davies Liu] SPARK-7836 and SPARK-7822: Python API of window functions

SPARK-7836 and SPARK-7822: Python API of window functions

778e2c0

rxin reviewed May 22, 2015
View reviewed changes

update docs

2649358

fix maxint in python 3

8936ade

rxin reviewed May 23, 2015
View reviewed changes

Davies Liu and others added 2 commits May 22, 2015 18:31

Merge branch 'master' of github.com:apache/spark into window4

ef55132

[SPARK-7322][SQL] Improve DataFrame window function documentation.

ed73cb4

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/functions.scala

davies force-pushed the window branch from 62bbb6c to c7c6859 Compare May 23, 2015 01:34

update docs

66092b4

davies force-pushed the window branch from c7c6859 to 66092b4 Compare May 23, 2015 01:35

rxin added a commit to rxin/spark that referenced this pull request May 23, 2015

Merge pull request apache#6364 from davies/window

7cb8985

[SPARK-7836] [SPARK-7822] Python API of window functions

rxin mentioned this pull request May 23, 2015

[SPARK-7322, SPARK-7836, SPARK-7822][SQL] DataFrame window function related updates #6374

Closed

davies closed this May 23, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7836] [SPARK-7822] Python API of window functions #6364

[SPARK-7836] [SPARK-7822] Python API of window functions #6364

davies commented May 22, 2015

rxin May 22, 2015

rxin May 22, 2015

SparkQA commented May 23, 2015

rxin commented May 23, 2015

SparkQA commented May 23, 2015

rxin May 23, 2015

davies commented May 23, 2015

SparkQA commented May 23, 2015

SparkQA commented May 23, 2015

SparkQA commented May 23, 2015

pwendell commented May 23, 2015

davies commented May 23, 2015

rxin commented May 23, 2015

[SPARK-7836] [SPARK-7822] Python API of window functions #6364

[SPARK-7836] [SPARK-7822] Python API of window functions #6364

Conversation

davies commented May 22, 2015

rxin May 22, 2015

Choose a reason for hiding this comment

rxin May 22, 2015

Choose a reason for hiding this comment

SparkQA commented May 23, 2015

rxin commented May 23, 2015

SparkQA commented May 23, 2015

rxin May 23, 2015

Choose a reason for hiding this comment

davies commented May 23, 2015

SparkQA commented May 23, 2015

SparkQA commented May 23, 2015

SparkQA commented May 23, 2015

pwendell commented May 23, 2015

davies commented May 23, 2015

rxin commented May 23, 2015