Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-21551][Python] Increase timeout for PythonRDD.serveIterator #19513

Closed
wants to merge 1 commit into from

Conversation

@FRosner
Copy link
Contributor

FRosner commented Oct 17, 2017

Backport of #18752 (https://issues.apache.org/jira/browse/SPARK-21551)

(cherry picked from commit 9d3c664)

@srowen
srowen approved these changes Oct 17, 2017
@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Oct 17, 2017

Test build #3950 has started for PR 19513 at commit 96b465b.

@HyukjinKwon

This comment has been minimized.

Copy link
Member

HyukjinKwon commented Oct 17, 2017

Hm, it seems because it also triggers the build with Python 2.6, and also due to a regression (SPARK-8763) by missing import subprocess.call - https://github.com/apache/spark/pull/9161/files#diff-af7df2796c411f6a78f3c32996c92b82R46

@HyukjinKwon

This comment has been minimized.

Copy link
Member

HyukjinKwon commented Oct 17, 2017

BTW, looks both #19514 and this builds were ran in the same amp-jenkins-worker-03.

@HyukjinKwon

This comment has been minimized.

Copy link
Member

HyukjinKwon commented Oct 17, 2017

cc @shaneknapp, would you mayybe have an idea about this? I suspect if this is the similar case we met before.

@shaneknapp

This comment has been minimized.

Copy link
Contributor

shaneknapp commented Oct 17, 2017

argh, it looks like the PATH variable got dropped, so it's not using the anaconda python install. i've marked worker-03 offline, and once the jobs there are done building i'll disconnect/reattach it. that'll fix this worker good.

@shaneknapp

This comment has been minimized.

Copy link
Contributor

shaneknapp commented Oct 17, 2017

after spot-checking the other workers, i'll need to restart the worker daemon on 02, 03, 07 and 08.

@HyukjinKwon

This comment has been minimized.

Copy link
Member

HyukjinKwon commented Oct 17, 2017

Wow, thanks for super fast action.

@HyukjinKwon

This comment has been minimized.

Copy link
Member

HyukjinKwon commented Oct 18, 2017

retest this please

@SparkQA

This comment has been minimized.

Copy link

SparkQA commented Oct 18, 2017

Test build #82884 has finished for PR 19513 at commit 96b465b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
asfgit pushed a commit that referenced this pull request Oct 20, 2017
Backport of #18752 (https://issues.apache.org/jira/browse/SPARK-21551)

(cherry picked from commit 9d3c664)

Author: peay <peay@protonmail.com>

Closes #19513 from FRosner/branch-2.1.
@srowen

This comment has been minimized.

Copy link
Member

srowen commented Oct 20, 2017

Merged to 2.1. @FRosner you'll have to close this manually

asfgit pushed a commit that referenced this pull request Oct 21, 2017
… explicit message for < Python 2.7

## What changes were proposed in this pull request?

Seems there was a mistake - missing import for `subprocess.call`, while refactoring this script a long ago, which should be used for backports of some missing functions in `subprocess`, specifically in < Python 2.7.

Reproduction is:

```
cd dev && python2.6
```

```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_call("ls")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sparktestsupport/shellutils.py", line 46, in subprocess_check_call
    retcode = call(*popenargs, **kwargs)
NameError: global name 'call' is not defined
```

For Jenkins logs, please see https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3950/console

Since we dropped the Python 2.6.x support, looks better we remove those workarounds and print out explicit error messages in order to reduce the efforts to find out the root causes for such cases, for example, `#19513 (comment).

## How was this patch tested?

Manually tested:

```
./dev/run-tests
```

```
Python versions prior to 2.7 are not supported.
```

```
./dev/run-tests-jenkins
```

```
Python versions prior to 2.7 are not supported.
```

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #19524 from HyukjinKwon/SPARK-22302.
@HyukjinKwon

This comment has been minimized.

Copy link
Member

HyukjinKwon commented Oct 22, 2017

@FRosner, close this one please too. This one is merged into 2.1 properly.

@FRosner FRosner closed this Oct 22, 2017
yaooqinn pushed a commit to yaooqinn/spark that referenced this pull request Sep 21, 2018
Backport of apache#18752 (https://issues.apache.org/jira/browse/SPARK-21551)

(cherry picked from commit 9d3c664)

Author: peay <peay@protonmail.com>

Closes apache#19513 from FRosner/branch-2.1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.