Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests #28085

Closed
wants to merge 31 commits into from
Closed

[SPARK-29641][PYTHON][CORE] Stage Level Sched: Add python api's and tests #28085

wants to merge 31 commits into from

Commits on Mar 30, 2020

  1. Stage level scheduling python api support

    @tgravescs
    tgravescs committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    acdf6e8 View commit details
    Browse the repository at this point in the history

  2. revert pom changes

    @tgravescs
    tgravescs committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    24c1a96 View commit details
    Browse the repository at this point in the history

  3. Fix log messages

    @tgravescs
    tgravescs committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    5647535 View commit details
    Browse the repository at this point in the history

  4. Try changing way we pass pyspark memory

    @tgravescs
    tgravescs committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    af69e4b View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2020

  1. Change to use local property to pass pyspark memory

    @tgravescs
    tgravescs committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    4a7f39a View commit details
    Browse the repository at this point in the history

  2. add missing api to get java map

    @tgravescs
    tgravescs committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    27e1a10 View commit details
    Browse the repository at this point in the history

  3. Add java api test

    @tgravescs
    tgravescs committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    af602b6 View commit details
    Browse the repository at this point in the history

  4. cleanup

    @tgravescs
    tgravescs committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    2b515c8 View commit details
    Browse the repository at this point in the history

  5. fix indentation

    @tgravescs
    tgravescs committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    a052427 View commit details
    Browse the repository at this point in the history

  6. fix newline around markup in python

    @tgravescs
    tgravescs committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    6a90fbe View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2020

  1. Update the version added for rdd api's

    @tgravescs
    tgravescs committed Apr 1, 2020
    Configuration menu
    Copy the full SHA
    2d754f7 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2020

  1. make java return values immutable

    @tgravescs
    tgravescs committed Apr 2, 2020
    Configuration menu
    Copy the full SHA
    1071f40 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2020

  1. Try reverting java api suite

    @tgravescs
    tgravescs committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    e81a480 View commit details
    Browse the repository at this point in the history

  2. Fix minor review comments

    @tgravescs
    tgravescs committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    ae8e312 View commit details
    Browse the repository at this point in the history

  3. Update to pass the executor cores into PythonRunner

    @tgravescs
    tgravescs committed Apr 6, 2020
    Configuration menu
    Copy the full SHA
    0d7c79f View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2020

  1. move python files to resources module and misc fixes

    @tgravescs
    tgravescs committed Apr 7, 2020
    Configuration menu
    Copy the full SHA
    956dc84 View commit details
    Browse the repository at this point in the history

  2. Fix how we pass the pyspark memory and cores

    @tgravescs
    tgravescs committed Apr 7, 2020
    Configuration menu
    Copy the full SHA
    32bca95 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2020

  1. Fix python imports

    @tgravescs
    tgravescs committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    e494c05 View commit details
    Browse the repository at this point in the history

  2. Fix java api suite test from hanging

    @tgravescs
    tgravescs committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    3e15ed9 View commit details
    Browse the repository at this point in the history

  3. change test to not rely on being able to set pyspark memory after 

    startup
    @tgravescs
    tgravescs committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    c3c885a View commit details
    Browse the repository at this point in the history

  4. Add in pyspark.resource module

    @tgravescs
    tgravescs committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    3562539 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2020

  1. review comments

    @tgravescs
    tgravescs committed Apr 13, 2020
    Configuration menu
    Copy the full SHA
    544119e View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2020

  1. Changes to allow using resource apis without SparkContext

    @tgravescs
    tgravescs committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    8469038 View commit details
    Browse the repository at this point in the history

  2. More changes to call without SparkContext

    @tgravescs
    tgravescs committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    a0b9137 View commit details
    Browse the repository at this point in the history

  3. cleanup

    @tgravescs
    tgravescs committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    2235654 View commit details
    Browse the repository at this point in the history

  4. fix style issues

    @tgravescs
    tgravescs committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    bf1a215 View commit details
    Browse the repository at this point in the history

  5. Change getResourceProfile to return None to match scala side

    @tgravescs
    tgravescs committed Apr 15, 2020
    Configuration menu
    Copy the full SHA
    62cb02c View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2020

  1. Change to make python versions do same thing as the scala versions as 

    hit some issues with switching between objects created before
SparkContext and ones after. Easier to understand this way. Add tests
    @tgravescs
    tgravescs committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    a6e9ac2 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2020

  1. add pyspark resource module to testing module

    @tgravescs
    tgravescs committed Apr 20, 2020
    Configuration menu
    Copy the full SHA
    528094c View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2020

  1. Update names of function/variable

    @tgravescs
    tgravescs committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    89be02e View commit details
    Browse the repository at this point in the history

  2. Other variable name changes

    @tgravescs
    tgravescs committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    354fb0c View commit details
    Browse the repository at this point in the history