[SPARK-34766][SQL] Do not capture maven config for views #31856

ulysses-you · 2021-03-17T02:11:41Z

What changes were proposed in this pull request?

Skip capture maven repo config for views.

Why are the changes needed?

Due to the bad network, we always use the thirdparty maven repo to run test. e.g.,

build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxxxx

It's failed with such error msg

[info] - show-tblproperties.sql *** FAILED *** (128 milliseconds)
[info] show-tblproperties.sql
[info] Expected "...rredTempViewNames [][]", but got "...rredTempViewNames [][
[info] view.sqlConfig.spark.sql.maven.additionalRemoteRepositories xxxxx]" Result did not match for query #6
[info] SHOW TBLPROPERTIES view (SQLQueryTestSuite.scala:464)

It's not necessary to capture the maven config to view since it's a session level config.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

manual test pass

build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxx

SparkQA · 2021-03-17T03:27:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40716/

SparkQA · 2021-03-17T03:36:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40716/

ulysses-you · 2021-03-17T08:56:42Z

cc @yaooqinn @maropu @cloud-fan

yaooqinn · 2021-03-17T09:12:16Z

LGTM

maropu · 2021-03-17T12:07:15Z

@yaooqinn Have you already tried to merge a commit? If not yet, please do it for your practice.

yaooqinn · 2021-03-17T12:11:15Z

Yes, I have taken my first shot with my own PR(which is a follow-up, and no need for the JIRA side). Let me merge this to test the full workflow.

yaooqinn · 2021-03-17T12:25:46Z

Thanks, Merged to master~

BTW, is the second run of the pr merge tool not able to change target branch

maropu · 2021-03-17T12:41:57Z

BTW, is the second run of the pr merge tool not able to change target branch

Why can't you select a target branch? It raises an error?

yaooqinn · 2021-03-17T13:12:28Z

A brief history w/ running ./dev/merge_spark_pr.py for the second round after this PR is in master

Which pull request would you like to merge? (e.g. 34): 31856

In this part, I was expecting it would ask me to enter the targeting branch, but it went to master directly

Pull request 31856 is not mergeable in its current form.
Continue? (experts only!) (y/n): y

=== Pull Request #31856 ===
title	[SPARK-34766][SQL] Do not capture maven config for views
source	ulysses-you/skip-maven-config
target	master
url	https://api.github.com/repos/apache/spark/pulls/31856

Proceed with merging pull request #31856? (y/n):
``

cloud-fan · 2021-03-17T13:17:21Z

after merging to master, the script will ask you if you want to merge it to other branches.

yaooqinn · 2021-03-17T14:14:58Z

after merging to master, the script will ask you if you want to merge it to other branches.

Yeah, I have seen this.

After if I reply n to it and I regret it, is there any chance for me to go back?

cloud-fan · 2021-03-17T14:26:27Z

No, now we need to create a new PR for 3.1...

yaooqinn · 2021-03-17T14:31:39Z

Er.. I was expecting something like

Hi @ulysses-you, sorry for my mistake, can we make a backport to 3.1

### What changes were proposed in this pull request? Skip capture maven repo config for views. ### Why are the changes needed? Due to the bad network, we always use the thirdparty maven repo to run test. e.g., ``` build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxxxx ``` It's failed with such error msg ``` [info] - show-tblproperties.sql *** FAILED *** (128 milliseconds) [info] show-tblproperties.sql [info] Expected "...rredTempViewNames [][]", but got "...rredTempViewNames [][ [info] view.sqlConfig.spark.sql.maven.additionalRemoteRepositories xxxxx]" Result did not match for query #6 [info] SHOW TBLPROPERTIES view (SQLQueryTestSuite.scala:464) ``` It's not necessary to capture the maven config to view since it's a session level config. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? manual test pass ``` build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxx ``` Closes apache#31856 from ulysses-you/skip-maven-config. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>

ulysses-you · 2021-03-18T06:58:15Z

created #31879

backport [#31856](#31856) for branch-3.1 ### What changes were proposed in this pull request? Skip capture maven repo config for views. ### Why are the changes needed? Due to the bad network, we always use the thirdparty maven repo to run test. e.g., ``` build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxxxx ``` It's failed with such error msg ``` [info] - show-tblproperties.sql *** FAILED *** (128 milliseconds) [info] show-tblproperties.sql [info] Expected "...rredTempViewNames [][]", but got "...rredTempViewNames [][ [info] view.sqlConfig.spark.sql.maven.additionalRemoteRepositories xxxxx]" Result did not match for query #6 [info] SHOW TBLPROPERTIES view (SQLQueryTestSuite.scala:464) ``` It's not necessary to capture the maven config to view since it's a session level config. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? manual test pass ``` build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxx ``` Closes #31856 from ulysses-you/skip-maven-config. Authored-by: ulysses-you <ulyssesyou18gmail.com> Signed-off-by: Kent Yao <yaoapache.org> Closes #31879 from ulysses-you/SPARK-34766-3-1. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

backport [apache#31856](apache#31856) for branch-3.1 ### What changes were proposed in this pull request? Skip capture maven repo config for views. ### Why are the changes needed? Due to the bad network, we always use the thirdparty maven repo to run test. e.g., ``` build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxxxx ``` It's failed with such error msg ``` [info] - show-tblproperties.sql *** FAILED *** (128 milliseconds) [info] show-tblproperties.sql [info] Expected "...rredTempViewNames [][]", but got "...rredTempViewNames [][ [info] view.sqlConfig.spark.sql.maven.additionalRemoteRepositories xxxxx]" Result did not match for query apache#6 [info] SHOW TBLPROPERTIES view (SQLQueryTestSuite.scala:464) ``` It's not necessary to capture the maven config to view since it's a session level config. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? manual test pass ``` build/sbt "test:testOnly *SQLQueryTestSuite" -Dspark.sql.maven.additionalRemoteRepositories=xxx ``` Closes apache#31856 from ulysses-you/skip-maven-config. Authored-by: ulysses-you <ulyssesyou18gmail.com> Signed-off-by: Kent Yao <yaoapache.org> Closes apache#31879 from ulysses-you/SPARK-34766-3-1. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

init

10f433e

github-actions bot added the SQL label Mar 17, 2021

key

b0937d9

yaooqinn approved these changes Mar 17, 2021

View reviewed changes

maropu approved these changes Mar 17, 2021

View reviewed changes

yaooqinn closed this in 48637a9 Mar 17, 2021

ulysses-you mentioned this pull request Mar 18, 2021

[SPARK-34766][SQL][3.1] Do not capture maven config for views #31879

Closed

ulysses-you deleted the skip-maven-config branch April 14, 2021 05:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-34766][SQL] Do not capture maven config for views #31856

[SPARK-34766][SQL] Do not capture maven config for views #31856

ulysses-you commented Mar 17, 2021

SparkQA commented Mar 17, 2021

SparkQA commented Mar 17, 2021

ulysses-you commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

maropu commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

yaooqinn commented Mar 17, 2021 •

edited

Loading

maropu commented Mar 17, 2021

yaooqinn commented Mar 17, 2021 •

edited

Loading

cloud-fan commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

cloud-fan commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

ulysses-you commented Mar 18, 2021

[SPARK-34766][SQL] Do not capture maven config for views #31856

[SPARK-34766][SQL] Do not capture maven config for views #31856

Conversation

ulysses-you commented Mar 17, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Mar 17, 2021

SparkQA commented Mar 17, 2021

ulysses-you commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

maropu commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

yaooqinn commented Mar 17, 2021 • edited Loading

maropu commented Mar 17, 2021

yaooqinn commented Mar 17, 2021 • edited Loading

A brief history w/ running ./dev/merge_spark_pr.py for the second round after this PR is in master

cloud-fan commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

cloud-fan commented Mar 17, 2021

yaooqinn commented Mar 17, 2021

ulysses-you commented Mar 18, 2021

yaooqinn commented Mar 17, 2021 •

edited

Loading

yaooqinn commented Mar 17, 2021 •

edited

Loading