-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-27405][SQL][TEST] Restrict the range of generated random timestamps #24316
Conversation
@dongjoon-hyun @srowen Please, take a look at the PR. |
Could you add a concrete example in the PR description for the following, too?
You may find some instance from the following failure.
|
Test build #104366 has finished for PR 24316 at commit
|
jenkins, retest this, please |
I would agree the PR is not tested by an explicit test but what would you like to test? I think we can trust to ScalaCheck's |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems OK to me. I think the point is that Arbitrary is going to pick values that aren't valid for this test? The range changes then as a fix, and we can see that tests pass before and after. What else?
Test build #104384 has finished for PR 24316 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What changes were proposed in this pull request?
In the PR, I propose to restrict the range of random timestamp literals generated in
LiteralGenerator. timestampLiteralGen
. The generator creates instances ofjava.sql.Timestamp
by passing milliseconds since epoch asLong
type. Converting the milliseconds to microseconds can cause arithmetic overflow of Long type because Catalyst's Timestamp type stores microseconds since epoch inLong
type internally as well. Proposed interval of random milliseconds is[Long.MinValue / 1000, Long.MaxValue / 1000]
.For example, generated timestamp
new java.sql.Timestamp(-3948373668011580000)
causesLong
overflow at the method:because
t.getTime()
returns-3948373668011580000
which is multiplied by1000
atMILLISECONDS.toMicros
, and the result-3948373668011580000000
is less thanLong.MinValue
.How was this patch tested?
By
DateExpressionsSuite
in the PR #24311