Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0 #19884

Closed
wants to merge 17 commits into from
Closed

[SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0 #19884

wants to merge 17 commits into from

Commits on Nov 27, 2017

  1. Upgrade Netty to 4.1.17

    @zsxwing @BryanCutler
    zsxwing authored and BryanCutler committed Nov 27, 2017
    Configuration menu
    Copy the full SHA
    2311fa2 View commit details
    Browse the repository at this point in the history

  2. Fix ProtocolSuite

    @zsxwing @BryanCutler
    zsxwing authored and BryanCutler committed Nov 27, 2017
    Configuration menu
    Copy the full SHA
    9d78299 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2017

  1. Add AbstractFileRegion

    @zsxwing @BryanCutler
    zsxwing authored and BryanCutler committed Dec 9, 2017
    Configuration menu
    Copy the full SHA
    de67ce2 View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2017

  1. initial code update for Arrow 0.8 after java refactor, most tests pas… 

    …sing still a couple issues remain
    @BryanCutler
    BryanCutler committed Dec 18, 2017
    Configuration menu
    Copy the full SHA
    1779dc3 View commit details
    Browse the repository at this point in the history

  2. updated JSON NaN to work with newer Jackson parser

    @BryanCutler
    BryanCutler committed Dec 18, 2017
    Configuration menu
    Copy the full SHA
    cbf6b97 View commit details
    Browse the repository at this point in the history

  3. fixed for latest Arrow master, removed Nullable prefix, removed vecto… 

    …rLayout in JSON
    @BryanCutler
    BryanCutler committed Dec 18, 2017
    Configuration menu
    Copy the full SHA
    feba0d0 View commit details
    Browse the repository at this point in the history

  4. fixed problems with type checking

    @BryanCutler
    BryanCutler committed Dec 18, 2017
    Configuration menu
    Copy the full SHA
    12812d8 View commit details
    Browse the repository at this point in the history

  5. removed casting workarounds in serializers, pyspark tests pass

    @BryanCutler
    BryanCutler committed Dec 18, 2017
    Configuration menu
    Copy the full SHA
    0590f8a View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2017

  1. no longer need to keep closed flag in toPayloadIterator

    @BryanCutler
    BryanCutler committed Dec 19, 2017
    Configuration menu
    Copy the full SHA
    d52b2af View commit details
    Browse the repository at this point in the history

  2. added requirement for minimum pyarrow version

    @BryanCutler
    BryanCutler committed Dec 19, 2017
    Configuration menu
    Copy the full SHA
    22c6b92 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2017

  1. moved pyarrow version check to sql.utils

    @BryanCutler
    BryanCutler committed Dec 20, 2017
    Configuration menu
    Copy the full SHA
    2149f72 View commit details
    Browse the repository at this point in the history

  2. change ArrowPythonRunner EOS to close the reader while keeping input … 

    …open
    @BryanCutler
    BryanCutler committed Dec 20, 2017
    Configuration menu
    Copy the full SHA
    0047f7a View commit details
    Browse the repository at this point in the history

  3. reworked udf pyarrow check

    @BryanCutler
    BryanCutler committed Dec 20, 2017
    Configuration menu
    Copy the full SHA
    d92ae90 View commit details
    Browse the repository at this point in the history

  4. make PandasUDFTests conditional on Pandas installed

    @BryanCutler
    BryanCutler committed Dec 20, 2017
    Configuration menu
    Copy the full SHA
    faa9f09 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2017

  1. need to skip doctest lines with pandas_udf

    @BryanCutler
    BryanCutler committed Dec 21, 2017
    Configuration menu
    Copy the full SHA
    423b68c View commit details
    Browse the repository at this point in the history

  2. workaround for timestamp casting

    @BryanCutler
    BryanCutler committed Dec 21, 2017
    Configuration menu
    Copy the full SHA
    ae84c84 View commit details
    Browse the repository at this point in the history

  3. revert doctest result

    @BryanCutler
    BryanCutler committed Dec 21, 2017
    Configuration menu
    Copy the full SHA
    b0200ef View commit details
    Browse the repository at this point in the history