Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-9026] [SPARK-4514] Modifications to JobWaiter, FutureAction, and AsyncRDDActions to support non-blocking operation #9264

Closed
wants to merge 29 commits into from
Closed

[SPARK-9026] [SPARK-4514] Modifications to JobWaiter, FutureAction, and AsyncRDDActions to support non-blocking operation #9264

wants to merge 29 commits into from

Commits on Oct 24, 2015

  1. reworked FutureAction and JobWaiter to avoid blocking or consuming th… 

    …reads while waiting for jobs
    @reggert
    reggert committed Oct 24, 2015
    Configuration menu
    Copy the full SHA
    4210aa6 View commit details
    Browse the repository at this point in the history

  2. reworked ComplexFutureAction and AsyncRDDActions.takeAsync to be non-… 

    …blocking
    @reggert
    reggert committed Oct 24, 2015
    Configuration menu
    Copy the full SHA
    1ad1abd View commit details
    Browse the repository at this point in the history

  3. corrected issues flagged by scalastyle

    @reggert
    reggert committed Oct 24, 2015
    Configuration menu
    Copy the full SHA
    de008ce View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2015

  1. added test to verify that FutureAction callback doesn't invoke Execut… 

    …ionContext too early
    @reggert
    reggert committed Nov 7, 2015
    Configuration menu
    Copy the full SHA
    17f6988 View commit details
    Browse the repository at this point in the history

  2. split out tests for SimpleFutureAction and ComplexFutureAction callba… 

    …ck handling
    @reggert
    reggert committed Nov 7, 2015
    Configuration menu
    Copy the full SHA
    4def989 View commit details
    Browse the repository at this point in the history

  3. merged conflicting import changes from upstream

    @reggert
    reggert committed Nov 7, 2015
    Configuration menu
    Copy the full SHA
    93b3065 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2015

  1. merged from upstream; resurrected accidentally deleted Utils class

    @reggert
    reggert committed Nov 10, 2015
    Configuration menu
    Copy the full SHA
    dcd2883 View commit details
    Browse the repository at this point in the history

  2. refactored duplicate code in SimpleFutureAction/ComplexFutureAction t… 

    …ests for non-blocking; cleaned up a few minor style issues in the process; added comment explaining why we're using Thread.sleep
    @reggert
    reggert committed Nov 10, 2015
    Configuration menu
    Copy the full SHA
    8624ef0 View commit details
    Browse the repository at this point in the history

  3. removed unused/redundant imports and cleaned up order of import state… 

    …ments
    @reggert
    reggert committed Nov 10, 2015
    Configuration menu
    Copy the full SHA
    31de51b View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2015

  1. removed unused implicit ExecutionContext parameter from ComplexFuture… 

    …Action.runJob
    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    0a6c614 View commit details
    Browse the repository at this point in the history

  2. added @DeveloperAPI annotation to SimpleFutureAction and ComplexFutur… 

    …eAction
    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    2c2da36 View commit details
    Browse the repository at this point in the history

  3. corrected order of imports

    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    cef7637 View commit details
    Browse the repository at this point in the history

  4. corrected minor style issue

    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    e364f93 View commit details
    Browse the repository at this point in the history

  5. reworked AsyncRDDActionsSuite.testAsyncAction to use semaphores inste… 

    …ad of Thread.sleep in order to reduce the potential for flakiness.
    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    0dd1128 View commit details
    Browse the repository at this point in the history

  6. cleaned up minor style issues

    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    50cd13e View commit details
    Browse the repository at this point in the history

  7. synchronized resultHandler call in JobWaiter.taskSucceeded

    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    7f4244d View commit details
    Browse the repository at this point in the history

  8. more style fixes; updated documentation for ComplexFutureAction.runJo… 

    …b, which has been renamed to submitJob
    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    dddad7f View commit details
    Browse the repository at this point in the history

  9. fixed a couple of other style issues

    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    19ef962 View commit details
    Browse the repository at this point in the history

  10. fixed issue flagged by scalastyle

    @reggert
    reggert committed Nov 11, 2015
    Configuration menu
    Copy the full SHA
    d1a905d View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2015

  1. added an assertion to verify that future completed properly

    @reggert
    reggert committed Nov 12, 2015
    Configuration menu
    Copy the full SHA
    c63ac17 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2015

  1. picked some nits

    @reggert
    reggert committed Nov 14, 2015
    Configuration menu
    Copy the full SHA
    489aabc View commit details
    Browse the repository at this point in the history

  2. picked another nit

    @reggert
    reggert committed Nov 14, 2015
    Configuration menu
    Copy the full SHA
    c19b3c0 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2015

  1. Refactored ComplexFutureAction to take a 'run' function as a construc… 

    …tor argument accepting a JobSubmitter, rather than having 'run' and 'submitJob' methods on ComplexFutureAction itself.
    @reggert
    reggert committed Nov 17, 2015
    Configuration menu
    Copy the full SHA
    601bb95 View commit details
    Browse the repository at this point in the history

  2. merged in changes from upstream master, including callsite preservation

    @reggert
    reggert committed Nov 17, 2015
    Configuration menu
    Copy the full SHA
    9da3533 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2015

  1. created reusable Smuggle construct to allow unserializable objects to… 

    … be 'smuggled' into tasks for testing purposes, and refactored AsyncRDDActionsSuite to make use of it
    @reggert
    reggert committed Nov 21, 2015
    Configuration menu
    Copy the full SHA
    a8ba899 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2015

  1. added unit test for SPARK-4514

    @reggert
    reggert committed Nov 22, 2015
    Configuration menu
    Copy the full SHA
    38b1442 View commit details
    Browse the repository at this point in the history

  2. Moved unit test for SPARK-4514 from AsyncRDDActionsSuite into StatusT… 

    …rackerSuite

This reverts commit 38b1442.
    @reggert
    reggert committed Nov 22, 2015
    Configuration menu
    Copy the full SHA
    5816489 View commit details
    Browse the repository at this point in the history

  3. fixed SPARK-4514 by calling setLocalProperties before submitting jobs… 

    … within takeAsync
    @reggert
    reggert committed Nov 22, 2015
    Configuration menu
    Copy the full SHA
    8fe8000 View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2015

  1. removed inaccurate comment line from AsyncRDDActionsSuite

    @reggert
    reggert committed Dec 13, 2015
    Configuration menu
    Copy the full SHA
    539ac43 View commit details
    Browse the repository at this point in the history