Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext #28784

Closed
wants to merge 5 commits into from
Closed

[SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext #28784

wants to merge 5 commits into from

Conversation

yaooqinn
Copy link
Member

What changes were proposed in this pull request?

Comparing to the long-running ThriftServer via start-script, we are more likely to hit the issue https://issues.apache.org/jira/browse/HIVE-10415 / https://issues.apache.org/jira/browse/SPARK-31626 in the developer API startWithContext

This PR apply SPARK-31626 to the developer API startWithContext

Why are the changes needed?

Fix the issue described in SPARK-31626

Does this PR introduce any user-facing change?

Yes, the hive scratch dir will be deleted if cleanup is enabled for calling startWithContext

How was this patch tested?

new test

@SparkQA
Copy link

SparkQA commented Jun 10, 2020

Test build #123766 has finished for PR 28784 at commit d055d60.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

juliuszsompolski
juliuszsompolski approved these changes Jun 12, 2020
View reviewed changes
Copy link
Contributor

@juliuszsompolski juliuszsompolski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

HyukjinKwon
HyukjinKwon approved these changes Jun 15, 2020
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. @yaooqinn can you rebase?

@yaooqinn
Copy link
Member Author

Thanks @HyukjinKwon and @juliuszsompolski, I was waiting for #28797 to be merged and then ping you guys. Now it's been done.
The conflicts disappear now. You can check again. Also cc @cloud-fan @maropu thanks

@yaooqinn yaooqinn changed the title [SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext Jun 15, 2020
@yaooqinn
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Jun 15, 2020

Test build #124035 has finished for PR 28784 at commit d055d60.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yaooqinn
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Jun 15, 2020

Test build #124037 has finished for PR 28784 at commit d055d60.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun mentioned this pull request Jun 15, 2020
Closed
@SparkQA
Copy link

SparkQA commented Jun 15, 2020

Test build #124059 has finished for PR 28784 at commit 1177e9a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class WidthBucket(
  • trait PredicateHelper extends Logging
  • case class ProcessingTimeTrigger(intervalMs: Long) extends Trigger
  • case class ContinuousTrigger(intervalMs: Long) extends Trigger

@maropu
Copy link
Member

maropu commented Jun 15, 2020

retest this please

@SparkQA
Copy link

SparkQA commented Jun 16, 2020

Test build #124079 has finished for PR 28784 at commit 1177e9a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class WidthBucket(
  • trait PredicateHelper extends Logging
  • case class ProcessingTimeTrigger(intervalMs: Long) extends Trigger
  • case class ContinuousTrigger(intervalMs: Long) extends Trigger

@HyukjinKwon
Copy link
Member

@yaooqinn, sorry can you rebase again? I caused a conflict before the tests finished.

@SparkQA
Copy link

SparkQA commented Jun 17, 2020

Test build #124169 has finished for PR 28784 at commit 2ff9573.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class TimeFormatters(date: DateFormatter, timestamp: TimestampFormatter)

@SparkQA
Copy link

SparkQA commented Jun 17, 2020

Test build #124170 has finished for PR 28784 at commit faf782e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yaooqinn yaooqinn changed the title [SPARK-31957][SQL][test-maven] Cleanup hive scratch dir for the developer api startWithContext [SPARK-31957][SQL][test-hive1.2] Cleanup hive scratch dir for the developer api startWithContext Jun 19, 2020
@yaooqinn
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Jun 19, 2020

Test build #124264 has finished for PR 28784 at commit 0b05d41.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class StateStoreConf(

@SparkQA
Copy link

SparkQA commented Jun 19, 2020

Test build #124262 has finished for PR 28784 at commit 0b05d41.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class StateStoreConf(

@yaooqinn
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Jun 19, 2020

Test build #124266 has finished for PR 28784 at commit 0b05d41.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class StateStoreConf(

@yaooqinn
Copy link
Member Author

retest this please

@yaooqinn yaooqinn changed the title [SPARK-31957][SQL][test-hive1.2] Cleanup hive scratch dir for the developer api startWithContext [SPARK-31957][SQL][test-hive1.2][test-maven] Cleanup hive scratch dir for the developer api startWithContext Jun 19, 2020
@yaooqinn
Copy link
Member Author

retest this please

1 similar comment
@yaooqinn
Copy link
Member Author

retest this please

@SparkQA
Copy link

SparkQA commented Jun 19, 2020

Test build #124274 has finished for PR 28784 at commit 0b05d41.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class StateStoreConf(

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-31957][SQL][test-hive1.2][test-maven] Cleanup hive scratch dir for the developer api startWithContext [SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext Jun 20, 2020
dongjoon-hyun
dongjoon-hyun approved these changes Jun 20, 2020
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you all!
Merged to master for Apache Spark 3.1.0 because this is an improvement.

@yaooqinn
Copy link
Member Author

Thank you all for reviewing and merging

rajatahujaatinmobi pushed a commit to rajatahujaatinmobi/spark-1 that referenced this pull request Jun 27, 2020
[SPARK-32021][SQL] Increase precision of seconds and fractions of `ma…
b42c3ed 
…ke_interval`

Change precision of seconds and its fraction from 8 to 18 to be able to construct intervals of max allowed microseconds value (long).

To improve UX of Spark SQL.

Yes

- Add tests to IntervalExpressionsSuite
- Add an example to the `MakeInterval` expression
- Add tests to `interval.sql`

Closes apache#28873 from MaxGekk/make_interval-sec-precision.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext

Comparing to the long-running ThriftServer via start-script, we are more likely to hit the issue https://issues.apache.org/jira/browse/HIVE-10415 / https://issues.apache.org/jira/browse/SPARK-31626 in the developer API `startWithContext`

This PR apply SPARK-31626 to the developer API `startWithContext`

Fix the issue described in  SPARK-31626

Yes, the hive scratch dir will be deleted if cleanup is enabled for calling `startWithContext`

new test

Closes apache#28784 from yaooqinn/SPARK-31957.

Authored-by: Kent Yao <yaooqinn@hotmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-32039][YARN][WEBUI][Master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

[SPARK-32039][YARN][WEBUI][Master] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

Merge branch 'ahujarajat261/SPARK-32039-change-yarn-webui-port-range-with-property-latest-spark' of https://github.com/rajatahujaatinmobi/spark-1 into ahujarajat261/SPARK-32039-change-yarn-webui-port-range-with-property-latest-spark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maropu maropu maropu approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

@juliuszsompolski juliuszsompolski juliuszsompolski approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
6 participants
@yaooqinn @SparkQA @maropu @HyukjinKwon @dongjoon-hyun @juliuszsompolski