[SPARK-21231] Disable installing pyarrow during run pip tests #18443
Conversation
|
are we planning on running the pyarrow tests in another script? |
|
So now that PyArrow is installed in one of the existing Python enviroments used for testing they should run automatically. I'll wait on the jenkins build to finish to verify that they are infact running in the current enviroment. |
|
excellent, thanks. |
There was a problem hiding this comment.
LGTM. Just one question though, now if pyarrow is not installed, then the tests will be skipped. That's good for users, but it would be better for Jenkins to give an error if it's not installed to make sure they are tested. Is there any way to configure this?
|
I don't remember if we specifically used python 3.5 here for the Conda tests because of arrow, and is that still required? |
|
@BryanCutler after looking at the code, it seems that it's definitely required for the for() loop located after the declaration, and to create the temporary python environment. |
|
Test build #78730 has finished for PR 18443 at commit
|
|
Jenkins retest this please |
|
Test build #78733 has finished for PR 18443 at commit
|
|
ugh, so another test failure is blocking this PR. I'm going to ask Jenkins retest this please. If this doesn't work, I'm going to back out the Arrow PR regardless until we get this sorted. |
|
hmm, twice an error with Kafka, third time's the charm! |
|
Jenkins retest this please |
|
Test build #78756 has finished for PR 18443 at commit
|
|
Closing since the source PR has been reverted and these changes should instead be merged into the updated new PR. |
What changes were proposed in this pull request?
Disable installing pyarrow during run pip tests, this was causing flakiness due to the conda-forge binstar -> conda migration and our out of date version of conda (which we still have).
How was this patch tested?
Now that pyarrow is installed in the py3k env we don't need to use the hack for pyarrow testing during the pip tests.
cc @BryanCutler @shaneknapp