Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-18143][SQL]Ignore Structured Streaming event logs to avoid breaking history server (branch 2.0) #15695

Closed
wants to merge 2 commits into from
Closed

[SPARK-18143][SQL]Ignore Structured Streaming event logs to avoid breaking history server (branch 2.0) #15695

wants to merge 2 commits into from

Conversation

zsxwing
Copy link
Member

@zsxwing zsxwing commented Oct 31, 2016

What changes were proposed in this pull request?

Backport #15663 to branch-2.0 and fixed conflicts in ReplayListenerBus.

How was this patch tested?

Jenkins

@zsxwing
Copy link
Member Author

zsxwing commented Oct 31, 2016

cc @tdas @rxin @marmbrus @yhuai

@rxin
Copy link
Contributor

rxin commented Oct 31, 2016

LGTM pending Jenkins.

JoshRosen
JoshRosen reviewed Oct 31, 2016
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala
// Structured Streaming in Spark 2.0.0.
// SparkListenerApplicationEnd is the only valid event and it's the last event. We use it
// to verify that we can skip broken jsons generated by Structured Streaming.
testReplayListenerBusWithBorkenEventJsons("query-event-logs-version-2.0.0.txt")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: Borken -> Broken

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: Borken -> Broken

Fixed

@SparkQA
Copy link

SparkQA commented Oct 31, 2016

Test build #67812 has finished for PR 15695 at commit e2d2bac.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 31, 2016

Test build #67815 has finished for PR 15695 at commit 0d4461a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing
Copy link
Member Author

zsxwing commented Oct 31, 2016

Looks like FileStreamSourceSuite is broken is 2.0. Looking at it.

@zsxwing
Copy link
Member Author

zsxwing commented Oct 31, 2016

Fixing the flaky test in #15699.

Retest this please

@SparkQA
Copy link

SparkQA commented Oct 31, 2016

Test build #3380 has finished for PR 15695 at commit 0d4461a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented Oct 31, 2016

Merging in branch-2.0. Can you close this?

@zsxwing
Copy link
Member Author

zsxwing commented Oct 31, 2016

Sure.

@zsxwing zsxwing closed this Oct 31, 2016
asfgit pushed a commit that referenced this pull request Oct 31, 2016
@zsxwing @rxin
[SPARK-18143][SQL] Ignore Structured Streaming event logs to avoid br…
300d596 
…eaking history server (branch 2.0)

## What changes were proposed in this pull request?

Backport #15663 to branch-2.0 and fixed conflicts in `ReplayListenerBus`.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #15695 from zsxwing/fix-event-log-2.0.
@zsxwing zsxwing deleted the fix-event-log-2.0 branch October 31, 2016 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JoshRosen JoshRosen JoshRosen left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zsxwing @rxin @SparkQA @JoshRosen