Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7735] [pyspark] Raise Exception on non-zero exit from pipe commands #6262

Closed
wants to merge 24 commits into from
Closed

[SPARK-7735] [pyspark] Raise Exception on non-zero exit from pipe commands #6262

wants to merge 24 commits into from

Commits on May 19, 2015

  1. Catch non-zero exit from pipe commands 

    This will allow problems with piped commands to be detected.
This will also allow tasks to be retried where errors are rare (such as network problems in piped commands).
    @megatron-me-uk
    megatron-me-uk committed May 19, 2015
    Configuration menu
    Copy the full SHA
    f552d49 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2015

  1. Remove space to fix style

    @megatron-me-uk
    megatron-me-uk committed Jun 3, 2015
    Configuration menu
    Copy the full SHA
    5745d85 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2015

  1. fix line too long style error

    @megatron-me-uk
    megatron-me-uk committed Jun 4, 2015
    Configuration menu
    Copy the full SHA
    45f4977 View commit details
    Browse the repository at this point in the history

  2. add space between words in multiline string

    @megatron-me-uk
    megatron-me-uk committed Jun 4, 2015
    Configuration menu
    Copy the full SHA
    0974f98 View commit details
    Browse the repository at this point in the history

  3. fix missing space around operator style 

    This is an error in PEP8 but not in pylint.
    @megatron-me-uk
    megatron-me-uk committed Jun 4, 2015
    Configuration menu
    Copy the full SHA
    1b3dc4e View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2015

  1. Add a test for rdd pipe functions

    @megatron-me-uk
    megatron-me-uk committed Jun 15, 2015
    Configuration menu
    Copy the full SHA
    8db4073 View commit details
    Browse the repository at this point in the history

  2. fix style issues in pipe test

    @megatron-me-uk
    megatron-me-uk committed Jun 15, 2015
    Configuration menu
    Copy the full SHA
    cc1a73d View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2015

  1. remove whitespace for style

    @megatron-me-uk
    megatron-me-uk committed Jun 18, 2015
    Configuration menu
    Copy the full SHA
    3ab8c7a View commit details
    Browse the repository at this point in the history

  2. wrap assertRaises with QuietTest

    @megatron-me-uk
    megatron-me-uk committed Jun 18, 2015
    Configuration menu
    Copy the full SHA
    3344a21 View commit details
    Browse the repository at this point in the history

  3. Pass a function handle to assertRaises 

    Also be more specific about the Exception we expect to see
    @megatron-me-uk
    megatron-me-uk committed Jun 18, 2015
    Configuration menu
    Copy the full SHA
    491d3fc View commit details
    Browse the repository at this point in the history

  4. fix list.sort returns None

    @megatron-me-uk
    megatron-me-uk committed Jun 18, 2015
    Configuration menu
    Copy the full SHA
    4153b02 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2015

  1. Chain generators to prevent potential deadlock

    @megatron-me-uk
    megatron-me-uk committed Jun 19, 2015
    Configuration menu
    Copy the full SHA
    8ed89a6 View commit details
    Browse the repository at this point in the history

  2. style fixes

    @megatron-me-uk
    megatron-me-uk committed Jun 19, 2015
    Configuration menu
    Copy the full SHA
    0486ae3 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2015

  1. make check_return_code an iterator

    @megatron-me-uk
    megatron-me-uk committed Jun 24, 2015
    Configuration menu
    Copy the full SHA
    8a9ef9c View commit details
    Browse the repository at this point in the history

  2. fix generator issue

    @megatron-me-uk
    megatron-me-uk committed Jun 24, 2015
    Configuration menu
    Copy the full SHA
    a0c0161 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2015

  1. add optional argument 'mode' for rdd.pipe 

    permissive - do not check returncode
strict - only allow returncode 0
grep - allow returncode 0 or 1
    @megatron-me-uk
    megatron-me-uk committed Jun 30, 2015
    Configuration menu
    Copy the full SHA
    34fcdc3 View commit details
    Browse the repository at this point in the history

  2. update rdd tests to test pipe modes

    @megatron-me-uk
    megatron-me-uk committed Jun 30, 2015
    Configuration menu
    Copy the full SHA
    a307d13 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #1 from megatron-me-uk/megatron-me-uk-patch-1 

    add optional argument 'mode' for rdd.pipe
    @megatron-me-uk
    megatron-me-uk committed Jun 30, 2015
    Configuration menu
    Copy the full SHA
    b0ac3a4 View commit details
    Browse the repository at this point in the history

  4. fix fail_condition 

    return True when an exception should be raised
    @megatron-me-uk
    megatron-me-uk committed Jun 30, 2015
    Configuration menu
    Copy the full SHA
    eb4801c View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2015

  1. Update rdd pipe tests for checkCode 

    (rather than mode)
    @megatron-me-uk
    megatron-me-uk committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    ab9a2e1 View commit details
    Browse the repository at this point in the history

  2. Update rdd pipe method for checkCode 

    use boolean checkCode rather than more complicated mode optional argument.

Also add param to docstring
    @megatron-me-uk
    megatron-me-uk committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    0c1e762 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #2 from megatron-me-uk/patch-4 

    Use boolean checkCode optional parameter
    @megatron-me-uk
    megatron-me-uk committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    574b564 View commit details
    Browse the repository at this point in the history

  4. fix blank line style error

    @megatron-me-uk
    megatron-me-uk committed Jul 7, 2015
    Configuration menu
    Copy the full SHA
    98fa101 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2015

  1. Remove spurious empty line

    @megatron-me-uk
    megatron-me-uk committed Jul 8, 2015
    Configuration menu
    Copy the full SHA
    04ae1d5 View commit details
    Browse the repository at this point in the history