-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-2729] [SQL] Forgot to match Timestamp type in ColumnBuilder #1636
Conversation
Can one of the admins verify this patch? |
Can you please add [SQL] to any PR title that touches Spark SQL? test this please |
We should also add a test case for this. |
jo, thanks, i will add [SQL] next time |
Will you have time to add a test case before Friday (the merge deadline for 1.1) or should I? |
it seems should be this one? i can do it |
|
if you have time, maybe take a look at #1612 ? it is about this ticket, https://issues.apache.org/jira/browse/SPARK-2710 , test suite just added. it will be a good feature for some use cases, such as we use RDBMS (OLTP) to store daily incremental data, and full data stored in Hadoop, we want to use Spark SQL to join or union the tables from these two data source. now we must hard code case class for each tables came from JDBC... and i find this one for CSV #1351 , it seems it is a good sample for adding new SchemaRDD types, need also to add some interface in sql.py and JavaSQLContext right? |
Thanks for working on this. I think that that test actually passes even without your change, so I don't think its sufficient. Maybe @liancheng has an idea of how to test this better. Regarding #1612, I'm very excited about this functionality as it is something we have been talking about for a while. I will look at it after we cut a release candidate for Spark 1.1 (this friday). |
@chutium Michael is right, To add a valid test case for this PR, I'd recommend |
thanks a lot to you both, cast string to timestamp is really clever :) |
Friendly reminder, it would be great to have a the test cases suggested by @liancheng so we can merge this by the deadline (friday). |
Thanks! I've merged this into master. |
This is a follow up of #1636. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes #1738 from liancheng/test-for-spark-2729 and squashes the following commits: b13692a [Cheng Lian] Added test case for SPARK-2729
This is a follow up of #1636. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes #1738 from liancheng/test-for-spark-2729 and squashes the following commits: b13692a [Cheng Lian] Added test case for SPARK-2729 (cherry picked from commit 866cf1f) Signed-off-by: Michael Armbrust <michael@databricks.com>
just a match forgot, found after SPARK-2710 , TimestampType can be used by a SchemaRDD generated from JDBC ResultSet Author: chutium <teng.qiu@gmail.com> Closes apache#1636 from chutium/SPARK-2729 and squashes the following commits: 71af77a [chutium] [SPARK-2729] [SQL] added Timestamp in NullableColumnAccessorSuite 39cf9f8 [chutium] [SPARK-2729] add Timestamp Type into ColumnBuilder TestSuite, ref. apache#1636 ab6ff97 [chutium] [SPARK-2729] Forgot to match Timestamp type in ColumnBuilder
This is a follow up of apache#1636. Author: Cheng Lian <lian.cs.zju@gmail.com> Closes apache#1738 from liancheng/test-for-spark-2729 and squashes the following commits: b13692a [Cheng Lian] Added test case for SPARK-2729
just a match forgot, found after SPARK-2710 , TimestampType can be used by a SchemaRDD generated from JDBC ResultSet