[SPARK-28200][SQL] Decimal overflow handling in ExpressionEncoder

[mickjermsurawong-stripe]

What changes were proposed in this pull request?

• Currently, ExpressionEncoder does not handle bigdecimal overflow. Round-tripping overflowing java/scala BigDecimal/BigInteger returns null.
  • The serializer encode java/scala BigDecimal to to sql Decimal, which still has the underlying data to the former.
  • When writing out to UnsafeRow, changePrecision will be false and row has null value.
    

    spark/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java

    Lines 202 to 206 in 24e1e41

    	if (input == null || !input.changePrecision(precision, scale)) {
    	BitSetMethods.set(getBuffer(), startingOffset, ordinal);
    	// keep the offset for future update
    	setOffsetAndSize(ordinal, 0);
    	} else {

• In SPARK-23179, an option to throw exception on decimal overflow was introduced.
• This PR adds the option in ExpressionEncoder to throw when detecting overflowing BigDecimal/BigInteger before its corresponding Decimal gets written to Row. This gives a consistent behavior between decimal arithmetic on sql expression (DecimalPrecision), and getting decimal from dataframe (RowEncoder)

Thanks to @mgaido91 for the very first PR SPARK-23179 and follow-up discussion on this change.
Thanks to @JoshRosen for working with me on this.

How was this patch tested?

added unit tests

[JoshRosen]

@mickjermsurawong-stripe, I think we also had a test case for RowEncoder? Could you also submit that change (in RowEncoderSuite.scala) as part of this PR? Even though we're not actually changing RowEncoder behavior here, the old code was lacking explicit test coverage for that path, so it'd be good to pull in that change to strengthen the tests

[JoshRosen]

jenkins this is ok to test

[mgaido91]

Seq("true", "false").foreach

[SparkQA]

Test build #107048 has finished for PR 25016 at commit b3740ba.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #107052 has finished for PR 25016 at commit 0ed62b3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

shall we use -BigDecimal... instead of .unary_-

[mgaido91]

mostly LGTM, just some style comments on the tests. Thanks.

Anyway, may you please also update the description of the PR? Because actually, the point here is not that we are not honoring the newly introduced flag, but previously we were not checking/handling at all the overflow in these situations. So I think we best should make this more clear in the PR description. Thanks.

[mgaido91]

Suggested change

	withSQLConf(SQLConf.DECIMAL_OPERATIONS_NULL_ON_OVERFLOW.key -> true.toString) {
	withSQLConf(SQLConf.DECIMAL_OPERATIONS_NULL_ON_OVERFLOW.key -> "true") {

[mgaido91]

Suggested change

	withSQLConf(SQLConf.DECIMAL_OPERATIONS_NULL_ON_OVERFLOW.key -> false.toString) {
	withSQLConf(SQLConf.DECIMAL_OPERATIONS_NULL_ON_OVERFLOW.key -> "false") {

[mgaido91]

nit: may you please also add the Jira number to the test case? Thanks.

[mickjermsurawong-stripe]

Updated. thank you Marco!

[SparkQA]

Test build #107062 has finished for PR 25016 at commit 2513ea7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #107081 has finished for PR 25016 at commit 87073da.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!