[SPARK-24812][SQL] Last Access Time in the table description is not valid #21775

sujith71955 · 2018-07-15T19:35:00Z

What changes were proposed in this pull request?

Last Access Time will always displayed wrong date Thu Jan 01 05:30:00 IST 1970 when user run DESC FORMATTED table command
In hive its displayed as "UNKNOWN" which makes more sense than displaying wrong date. seems to be a limitation as of now even from hive, better we can follow the hive behavior unless the limitation has been resolved from hive.

spark client output

Hive client output

How was this patch tested?

UT has been added which makes sure that the wrong date "Thu Jan 01 05:30:00 IST 1970 "
shall not be added as value for the Last Access property

xuanyuanking · 2018-07-16T14:25:00Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+      val lastAcessField = desc.filter((r: Row) => r.getValuesMap(Seq("col_name"))
+        .get("col_name").getOrElse("").equals("Last Access"))
+      // Check whether lastAcessField key is exist
+      assert(!lastAcessField.isEmpty)


lastAccessField.nonEmpty

xuanyuanking · 2018-07-16T14:25:16Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+      sql(s"create table" +
+        s" if not exists t1 (c1_int int, c2_string string, c3_float float)")
+      val desc = sql("DESC FORMATTED t1").collect().toSeq
+      val lastAcessField = desc.filter((r: Row) => r.getValuesMap(Seq("col_name"))


nit: lastAccessField

xuanyuanking · 2018-07-16T14:28:28Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala

+    val lastAccess = {
+      if (-1 == lastAccessTime) "UNKNOWN" else new Date(lastAccessTime).toString
+    }
+    map.put("Last Access", lastAccess)


No need for the val lastAccess?

map.put("Last Access", if (-1 == lastAccessTime) "UNKNOWN" else new Date(lastAccessTime).toString)

the current way is also fine

xuanyuanking · 2018-07-16T14:30:39Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+        .get("col_name").getOrElse("").equals("Last Access"))
+      // Check whether lastAcessField key is exist
+      assert(!lastAcessField.isEmpty)
+      val validLastAcessFieldValue = lastAcessField.filterNot((r: Row) => ((r


where is the val validLastAcessFieldValue used?

xuanyuanking · 2018-07-16T14:31:25Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+      val validLastAcessFieldValue = lastAcessField.filterNot((r: Row) => ((r
+        .getValuesMap(Seq("data_type"))
+        .get("data_type").contains(new Date(-1).toString))))
+      assert(lastAcessField.size!=0)


code style nit: blank before and after '!='

HyukjinKwon · 2018-07-20T01:54:47Z

ok to test

HyukjinKwon · 2018-07-20T01:55:22Z

Seems making sense.

seems to be a limitation as of now even from hive, better we can follow the hive behavior unless the limitation has been resolved from hive.

Do you maybe know related Hive side ticket or can you point me out the related codes?

HyukjinKwon · 2018-07-20T01:56:47Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+  test("desc formatted table for last access verification") {
+    withTable("t1") {
+      sql(s"create table" +
+        s" if not exists t1 (c1_int int, c2_string string, c3_float float)")


nit:

sql( "CREATE TABLE IF NOT EXISTS t1 (c1_int INT, c2_string STRING, c3_float FLOAT)")

HyukjinKwon · 2018-07-20T02:01:43Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+        s" if not exists t1 (c1_int int, c2_string string, c3_float float)")
+      val desc = sql("DESC FORMATTED t1").collect().toSeq
+      val lastAccessField = desc.filter((r: Row) => r.getValuesMap(Seq("col_name"))
+        .get("col_name").getOrElse("").equals("Last Access"))


Can we simplify this via, for instance, desc.filter($"col_name".startswith("...")).select("data_type")?

HyukjinKwon · 2018-07-20T02:02:27Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

@@ -2250,6 +2251,22 @@ class HiveDDLSuite
    }
  }

+  test("desc formatted table for last access verification") {


let's name it SPARK-24812: desc formatted table for last access verification

SparkQA · 2018-07-20T06:05:03Z

Test build #93311 has finished for PR 21775 at commit b527fdc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-07-20T16:39:05Z

This is an external change. Please add one more point in the migration guide.

sujith71955 · 2018-07-20T18:31:50Z

sure, i will update the PR based on the comments, Thanks for suggestions.

SparkQA · 2018-07-22T11:34:52Z

Test build #93406 has finished for PR 21775 at commit e9b0f91.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-07-22T11:44:51Z

Test build #93407 has finished for PR 21775 at commit 2008c21.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

sujith71955 · 2018-07-22T11:46:30Z

@HyukjinKwon
seems to be a limitation as of now even from hive, better we can follow the hive behavior unless the limitation has been resolved from hive.

Hive-2526 is the JIRA related to this issue , could not figure out any other open JIRA's mentioning this problem

SparkQA · 2018-07-22T16:01:41Z

Test build #93409 has finished for PR 21775 at commit 140c4ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-07-23T01:19:12Z

retest this please

SparkQA · 2018-07-23T05:29:41Z

Test build #93421 has finished for PR 21775 at commit 140c4ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2018-07-23T05:35:04Z

docs/sql-programming-guide.md

@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see

 ## Upgrading From Spark SQL 2.3 to 2.4

+  - Since Spark 2.4, Spark will display hive table description column  `Last Access` value  as `UNKNOWN` following the Hive system.


This is applicable to both native and hive tables. How about changing it to

Spark will display table description column Last Access value as UNKNOWN when the value was Jan 01 1970.

Right, its applicable for both type, i will update the message as per your comment. Thanks

SparkQA · 2018-07-23T07:05:01Z

Test build #93431 has finished for PR 21775 at commit 76a34c6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2018-07-23T07:22:13Z

ok to test

HyukjinKwon · 2018-07-23T07:22:20Z

retest this please

SparkQA · 2018-07-23T11:34:52Z

Test build #93437 has finished for PR 21775 at commit 76a34c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…alid ## What changes were proposed in this pull request? Last Access Time will always displayed wrong date Wed Dec 31 15:59:59 PST 1969 when user run DESC FORMATTED table command In hive its displayed as "UNKNOWN" which makes more sense than displaying wrong date. seems to be a limitation as of now, better we can follow the hive behavior unless the limitation has been resolved from hive. ## How was this patch tested? UT has been added which makes sure that the wrong date "Wed Dec 31 15:59:59 PST 1969 " shall not be added as value for the Last Access property

sujith71955 · 2018-07-24T07:14:18Z

@HyukjinKwon @gatorsmile All issues has been addressed, please let me know how this patch looks like. Thanks .

gatorsmile · 2018-07-24T18:30:15Z

retest this please

gatorsmile · 2018-07-24T18:30:46Z

The commit has been tested.

LGTM Thanks! Merged to master.

SparkQA · 2018-07-24T22:43:23Z

Test build #93513 has finished for PR 21775 at commit 76a34c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

xuanyuanking reviewed Jul 16, 2018

View reviewed changes

sujith71955 force-pushed the master_hive branch from 44e489d to b527fdc Compare July 18, 2018 18:02

HyukjinKwon reviewed Jul 20, 2018

View reviewed changes

sujith71955 force-pushed the master_hive branch from b527fdc to e9b0f91 Compare July 22, 2018 11:31

sujith71955 force-pushed the master_hive branch from e9b0f91 to 2008c21 Compare July 22, 2018 11:40

sujith71955 force-pushed the master_hive branch from 2008c21 to 140c4ce Compare July 22, 2018 11:50

gatorsmile reviewed Jul 23, 2018

View reviewed changes

sujith71955 force-pushed the master_hive branch from 140c4ce to 76a34c6 Compare July 23, 2018 05:50

asfgit closed this in d4a277f Jul 24, 2018

srowen mentioned this pull request Sep 16, 2019

[SPARK-28930][SQL] Last Access Time value shall display 'UNKNOWN' in all clients #25720

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-24812][SQL] Last Access Time in the table description is not valid #21775

[SPARK-24812][SQL] Last Access Time in the table description is not valid #21775

sujith71955 commented Jul 15, 2018 •

edited

xuanyuanking Jul 16, 2018

xuanyuanking Jul 16, 2018

xuanyuanking Jul 16, 2018

gatorsmile Jul 23, 2018

xuanyuanking Jul 16, 2018

xuanyuanking Jul 16, 2018

HyukjinKwon commented Jul 20, 2018

HyukjinKwon commented Jul 20, 2018

HyukjinKwon Jul 20, 2018 •

edited

HyukjinKwon Jul 20, 2018

HyukjinKwon Jul 20, 2018

SparkQA commented Jul 20, 2018

gatorsmile commented Jul 20, 2018

sujith71955 commented Jul 20, 2018

SparkQA commented Jul 22, 2018

SparkQA commented Jul 22, 2018

sujith71955 commented Jul 22, 2018

SparkQA commented Jul 22, 2018

HyukjinKwon commented Jul 23, 2018

SparkQA commented Jul 23, 2018

gatorsmile Jul 23, 2018

sujith71955 Jul 23, 2018

sujith71955 Jul 23, 2018

SparkQA commented Jul 23, 2018

HyukjinKwon commented Jul 23, 2018

HyukjinKwon commented Jul 23, 2018

SparkQA commented Jul 23, 2018

sujith71955 commented Jul 24, 2018

gatorsmile commented Jul 24, 2018

gatorsmile commented Jul 24, 2018

SparkQA commented Jul 24, 2018

		@@ -1843,6 +1843,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see

		## Upgrading From Spark SQL 2.3 to 2.4

		- Since Spark 2.4, Spark will display hive table description column `Last Access` value as `UNKNOWN` following the Hive system.

[SPARK-24812][SQL] Last Access Time in the table description is not valid #21775

[SPARK-24812][SQL] Last Access Time in the table description is not valid #21775

Conversation

sujith71955 commented Jul 15, 2018 • edited

What changes were proposed in this pull request?

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon commented Jul 20, 2018

HyukjinKwon commented Jul 20, 2018

HyukjinKwon Jul 20, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 20, 2018

gatorsmile commented Jul 20, 2018

sujith71955 commented Jul 20, 2018

SparkQA commented Jul 22, 2018

SparkQA commented Jul 22, 2018

sujith71955 commented Jul 22, 2018

SparkQA commented Jul 22, 2018

HyukjinKwon commented Jul 23, 2018

SparkQA commented Jul 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 23, 2018

HyukjinKwon commented Jul 23, 2018

HyukjinKwon commented Jul 23, 2018

SparkQA commented Jul 23, 2018

sujith71955 commented Jul 24, 2018

gatorsmile commented Jul 24, 2018

gatorsmile commented Jul 24, 2018

SparkQA commented Jul 24, 2018

sujith71955 commented Jul 15, 2018 •

edited

HyukjinKwon Jul 20, 2018 •

edited