[SPARK-14321][SQL] Reduce date format cost and string-to-date cost in date functions

[rajeshbalamohan]

What changes were proposed in this pull request?

Here is the generated code snippet when executing date functions. SimpleDateFormat is fairly expensive and can show up bottleneck when processing millions of records. It would be better to instantiate it once.

/* 066 */     UTF8String primitive5 = null;
/* 067 */     if (!isNull4) {
/* 068 */       try {
/* 069 */         primitive5 = UTF8String.fromString(new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(
/* 070 */             new java.util.Date(primitive7 * 1000L)));
/* 071 */       } catch (java.lang.Throwable e) {
/* 072 */         isNull4 = true;
/* 073 */       }
/* 074 */     }

With modified code, here is the generated code

/* 010 */   private java.text.SimpleDateFormat sdf2;
/* 011 */   private UnsafeRow result13;
/* 012 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder bufferHolder14;
/* 013 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter15;
/* 014 */
...
...
/* 065 */     boolean isNull0 = isNull3;
/* 066 */     UTF8String primitive1 = null;
/* 067 */     if (!isNull0) {
/* 068 */       try {
/* 069 */         if (sdf2 == null) {
/* 070 */           sdf2 = new java.text.SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
/* 071 */         }
/* 072 */         primitive1 = UTF8String.fromString(sdf2.format(
/* 073 */             new java.util.Date(primitive4 * 1000L)));
/* 074 */       } catch (java.lang.Throwable e) {
/* 075 */         isNull0 = true;
/* 076 */       }
/* 077 */     }

Similarly Calendar.getInstance was used in DateTimeUtils which can be lazily inited.

How was this patch tested?

org.apache.spark.sql.catalyst.expressions.DateExpressionsSuite,org.apache.spark.sql.catalyst.util.DateTimeUtilsSuite

[rajeshbalamohan]

@cloud-fan - Sorry about the delay. Rebased SPARK-14321 for master. #12105 had become stale and got little messy in my system. Ended up creating this PR. I will close the earlier one after review.

[yangw1234]

This is trivial but why use a different variable name here from the above one? (which is called "formatter")

[cloud-fan]

formatter looks better

[SparkQA]

Test build #60042 has finished for PR 13522 at commit 602d4a7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

LGTM except one naming comment, thanks for working on it!

[rajeshbalamohan]

Thank you. I have pushed the fixes in the recent commit.

[SparkQA]

Test build #60118 has finished for PR 13522 at commit 425aa7e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

Some more ideas: I think we should create a subclass of UnixTime to handle the left.dataType == StringType && right.foldable case. In optimizer, we can replace it with null literal if the formatter is null, then we don't need to do null check for every record.