[SPARK-29991][INFRA] Support test-hive1.2 in PR Builder
#26695
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongjoon-hyun
changed the title
[test-hive1.2] in PR Builder[test-hive1.2] in PR Builder
dongjoon-hyun
changed the title
[test-hive1.2] in PR Builder[test-hive1.2] in PR Builder
dongjoon-hyun
changed the title
[test-hive1.2] in PR Buildertest-hive1.2 in PR Builder
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dongjoon-hyun
changed the title
test-hive1.2 in PR Buildertest hive1.2 in PR Builder
|
Retest this please. |
dongjoon-hyun
changed the title
test hive1.2 in PR Buildertest-hive1.2 in PR Builder
This comment has been minimized.
This comment has been minimized.
|
Test build #114550 has finished for PR 26695 at commit
|
|
Test build #114551 has finished for PR 26695 at commit
|
|
Hi, @gatorsmile , @srowen , @HyukjinKwon . |
srowen
approved these changes
Nov 28, 2019
|
Thank you for review and approval, @srowen . Yes. Right. |
|
Merged to master. |
HyukjinKwon
reviewed
Nov 29, 2019
| } | ||
|
|
||
| if hadoop_version in sbt_maven_hadoop_profiles: | ||
| return sbt_maven_hadoop_profiles[hadoop_version] | ||
| if ("ghprbPullTitle" in os.environ and |
There was a problem hiding this comment.
Can we do it consistently with what we handle Hadoop versions? (e.g., 8ab1306#diff-3c9f4fccf7d30ce2e8fa86db2ad1fdad)
There was a problem hiding this comment.
and .. seems like we should change c98e5eb#diff-180360612c6b8c4ed830919bbb4dd459R56 too if this PR also targets to change default Hive profile to 2.3.
|
Seems this PR caused the test failure - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114582/testReport/ It looks like the tests against Hive 2.3 didn't run because the PR title constantly has After reverting this, seems the tests were recovered (#26706) |
|
Since this change blocks other PRs, let me revert this one for now. |
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Dec 6, 2019
### What changes were proposed in this pull request? Currently, Apache Spark PR Builder using `hive-1.2` for `hadoop-2.7` and `hive-2.3` for `hadoop-3.2`. This PR aims to support `[test-hive1.2]` in PR Builder in order to cut the correlation between `hive-1.2/2.3` to `hadoop-2.7/3.2`. After this PR, the PR Builder will use `hive-2.3` by default for all profiles (if there is no `test-hive1.2`.) ### Why are the changes needed? This new tag allows us more flexibility. ### Does this PR introduce any user-facing change? No. (This is a dev-only change.) ### How was this patch tested? Check the Jenkins triggers in this PR. **BEFORE** ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-1.2 -Pyarn -Pkubernetes -Phive -Phadoop-cloud -Pspark-ganglia-lgpl -Phive-thriftserver -Pkinesis-asl -Pmesos test:package streaming-kinesis-asl-assembly/assembly ``` **AFTER** 1. Title: [[SPARK-29991][INFRA][test-hive1.2] Support `test-hive1.2` in PR Builder](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114550/testReport) ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-1.2 -Pkinesis-asl -Phadoop-cloud -Pyarn -Phive -Pmesos -Pspark-ganglia-lgpl -Pkubernetes -Phive-thriftserver test:package streaming-kinesis-asl-assembly/assembly ``` 2. Title: [[SPARK-29991][INFRA] Support `test hive1.2` in PR Builder](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/114551/testReport) - Note that I removed the hyphen intentionally from `test-hive1.2`. ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-thriftserver -Pkubernetes -Pspark-ganglia-lgpl -Phadoop-cloud -Phive -Pmesos -Pyarn -Pkinesis-asl test:package streaming-kinesis-asl-assembly/assembly ``` Closes apache#26695 from dongjoon-hyun/SPARK-29991. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, Apache Spark PR Builder using
hive-1.2forhadoop-2.7andhive-2.3forhadoop-3.2. This PR aims to support[test-hive1.2]in PR Builder in order to cut the correlation betweenhive-1.2/2.3tohadoop-2.7/3.2. After this PR, the PR Builder will usehive-2.3by default for all profiles (if there is notest-hive1.2.)Why are the changes needed?
This new tag allows us more flexibility.
Does this PR introduce any user-facing change?
No. (This is a dev-only change.)
How was this patch tested?
Check the Jenkins triggers in this PR.
BEFORE
AFTER
test-hive1.2in PR Buildertest hive1.2in PR Buildertest-hive1.2.