Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-21190][PYSPARK] Python Vectorized UDFs #18659

Closed
wants to merge 19 commits into from
Closed

[SPARK-21190][PYSPARK] Python Vectorized UDFs #18659

wants to merge 19 commits into from

Commits on Sep 1, 2017

  1. vectorized udfs working but hardcoded for ArrowPandasSerializer

    @BryanCutler
    BryanCutler committed Sep 1, 2017
    Configuration menu
    Copy the full SHA
    be81ef6 View commit details
    Browse the repository at this point in the history

  2. Added conf for enabling vectorized UDFs, now working

    @BryanCutler
    BryanCutler committed Sep 1, 2017
    Configuration menu
    Copy the full SHA
    8569736 View commit details
    Browse the repository at this point in the history

  3. Columns for ArrowPandasSerializer need unique name

    @BryanCutler
    BryanCutler committed Sep 1, 2017
    Configuration menu
    Copy the full SHA
    9236e99 View commit details
    Browse the repository at this point in the history

  4. fixup ArrowEvalPythonExec to work with new fromPayloadIterator

    @BryanCutler
    BryanCutler committed Sep 1, 2017
    Configuration menu
    Copy the full SHA
    cc7ed5a View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2017

  1. changed conf to use a vectorized flag instead on per UDF basis

    @BryanCutler
    BryanCutler committed Sep 6, 2017
    Configuration menu
    Copy the full SHA
    cf764b0 View commit details
    Browse the repository at this point in the history

  2. fixed chaining of multiple udfs in serializer

    @BryanCutler
    BryanCutler committed Sep 6, 2017
    Configuration menu
    Copy the full SHA
    4f6c950 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2017

  1. changed decorator to pandas_udf

    @BryanCutler
    BryanCutler committed Sep 12, 2017
    Configuration menu
    Copy the full SHA
    91dead2 View commit details
    Browse the repository at this point in the history

  2. changed type to SQL_PANDAS_UDF

    @BryanCutler
    BryanCutler committed Sep 12, 2017
    Configuration menu
    Copy the full SHA
    4a2fec2 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2017

  1. Add tests. 

    Modify test_vectorized_udf_datatype_string not to fail by unrelated error.

closes #26
    @ueshin @BryanCutler
    ueshin authored and BryanCutler committed Sep 13, 2017
    Configuration menu
    Copy the full SHA
    518126e View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2017

  1. revert unrelated change

    @BryanCutler
    BryanCutler committed Sep 14, 2017
    Configuration menu
    Copy the full SHA
    3b4465c View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2017

  1. added optional kwarg to pandas_udf to get expected return vector length

    @BryanCutler
    BryanCutler committed Sep 15, 2017
    Configuration menu
    Copy the full SHA
    25e3a71 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2017

  1. style fixes

    @BryanCutler
    BryanCutler committed Sep 18, 2017
    Configuration menu
    Copy the full SHA
    dc237e7 View commit details
    Browse the repository at this point in the history

  2. changed kwarg to use length instead of size key

    @BryanCutler
    BryanCutler committed Sep 18, 2017
    Configuration menu
    Copy the full SHA
    4a0691b View commit details
    Browse the repository at this point in the history

  3. Added pyarrow.Array type to coerce promoted types, working with pyarr… 

    …ow 0.7
    @BryanCutler
    BryanCutler committed Sep 18, 2017
    Configuration menu
    Copy the full SHA
    d49a3db View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2017

  1. Refactor EvalPythonExec.

    @ueshin
    ueshin committed Sep 19, 2017
    Configuration menu
    Copy the full SHA
    69112a5 View commit details
    Browse the repository at this point in the history

  2. fixes from comments in PR

    @BryanCutler
    BryanCutler committed Sep 19, 2017
    Configuration menu
    Copy the full SHA
    f451d65 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2017

  1. use generator for conforming series input in serializer

    @BryanCutler
    BryanCutler committed Sep 20, 2017
    Configuration menu
    Copy the full SHA
    44a20f6 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2017

  1. use fillna before casting with astype

    @BryanCutler
    BryanCutler committed Sep 21, 2017
    Configuration menu
    Copy the full SHA
    53926cc View commit details
    Browse the repository at this point in the history

  2. added test and fix for chained pandas_udf

    @BryanCutler
    BryanCutler committed Sep 21, 2017
    Configuration menu
    Copy the full SHA
    b8ffa50 View commit details
    Browse the repository at this point in the history