New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-23565] [SS] New error message for structured streaming sources assertion #20946
Conversation
…r a query changes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change the title to '[SPARK-23565][SS]...' is more accurate.
@@ -39,7 +39,9 @@ case class OffsetSeq(offsets: Seq[Option[Offset]], metadata: Option[OffsetSeqMet | |||
* cannot be serialized). | |||
*/ | |||
def toStreamProgress(sources: Seq[BaseStreamingSource]): StreamProgress = { | |||
assert(sources.size == offsets.size) | |||
assert(sources.size == offsets.size, s"There are [${offsets.size}] sources in the " + | |||
s"checkpoint offsets and now there are [${sources.size}] sources requested by the query. " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: extra blank in the end
@@ -125,6 +125,19 @@ class OffsetSeqLogSuite extends SparkFunSuite with SharedSQLContext { | |||
assert(offsetSeq.metadata === Some(OffsetSeqMetadata(0L, 1480981499528L))) | |||
} | |||
|
|||
test("assertion that number of checkpoint offsets match number of sources") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe no need to add UT for the log change.
Hi Xuan, I have updated the title, removed the extra white space after the period / dot and removed the UT. |
Hi @xuanyuanking, please see comments above. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 To me, useful in SS debugging, cc @cloud-fan to take a look.
cc @zsxwing |
okey to test LGTM |
ok to test |
Test build #89881 has finished for PR 20946 at commit
|
Thanks! Merging to master. |
What changes were proposed in this pull request?
A more informative message to tell you why a structured streaming query cannot continue if you have added more sources, than there are in the existing checkpoint offsets.
How was this patch tested?
I added a Unit Test.