Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-6967] [SQL] fix date type convertion in jdbcrdd #5590

Closed
wants to merge 2 commits into from

Conversation

adrian-wang
Copy link
Contributor

This pr convert java.sql.Date type into Int for JDBCRDD.

@SparkQA
Copy link

SparkQA commented Apr 20, 2015

Test build #30581 has finished for PR 5590 at commit 3c9184c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@marmbrus
Copy link
Contributor

Could we add a unit test in JDBCSuite?

@SparkQA
Copy link

SparkQA commented Apr 22, 2015

Test build #30727 has finished for PR 5590 at commit f897b81.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@rxin
Copy link
Contributor

rxin commented Apr 23, 2015

Merging in master and branch-1.3.

@asfgit asfgit closed this in 04525c0 Apr 23, 2015
asfgit pushed a commit that referenced this pull request Apr 23, 2015
This pr convert java.sql.Date type into Int for JDBCRDD.

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes #5590 from adrian-wang/datebug and squashes the following commits:

f897b81 [Daoyuan Wang] add a test case
3c9184c [Daoyuan Wang] fix date type convertion in jdbcrdd

(cherry picked from commit 04525c0)
Signed-off-by: Reynold Xin <rxin@databricks.com>

Conflicts:
	sql/core/src/main/scala/org/apache/spark/sql/jdbc/JDBCRDD.scala
markhamstra pushed a commit to markhamstra/spark that referenced this pull request Apr 23, 2015
This pr convert java.sql.Date type into Int for JDBCRDD.

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes apache#5590 from adrian-wang/datebug and squashes the following commits:

f897b81 [Daoyuan Wang] add a test case
3c9184c [Daoyuan Wang] fix date type convertion in jdbcrdd

(cherry picked from commit 04525c0)
Signed-off-by: Reynold Xin <rxin@databricks.com>

Conflicts:
	sql/core/src/main/scala/org/apache/spark/sql/jdbc/JDBCRDD.scala
@nadavoosh
Copy link

hi @adrian-wang ! I am using spark and needed to include this fix, since I am reading from a table that has Date types. I rebuilt spark with this change & it works well.

I just ran into a new problem though: when the Date field has null values, spark throws a java.lang.NullPointerException
at org.apache.spark.sql.types.DateUtils$.javaDateToDays(DateUtils.scala:39)
error. Any ideas on how I can fix that?

Thank you for your help!

@adrian-wang
Copy link
Contributor Author

@nadavoosh Thanks for your feedback! I will take a look at this soon.

@nadavoosh
Copy link

@adrian-wang this worked for me:

        case DateConversion       => {
                                      val   x = rs.getDate(pos)
                                       mutableRow.update(i, if (x == null) null else DateUtils.fromJavaDate(x) )
                                     }
        case DecimalConversion    => {
                                      val x = rs.getBigDecimal(pos)
                                      mutableRow.update(i, if (x == null) null else Decimal(x) )
                                     }

asfgit pushed a commit that referenced this pull request May 7, 2015
Thank nadavoosh point this out in #5590

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes #5877 from adrian-wang/jdbcrdd and squashes the following commits:

cc11900 [Daoyuan Wang] avoid NPE in jdbcrdd
asfgit pushed a commit that referenced this pull request May 7, 2015
Thank nadavoosh point this out in #5590

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes #5877 from adrian-wang/jdbcrdd and squashes the following commits:

cc11900 [Daoyuan Wang] avoid NPE in jdbcrdd

(cherry picked from commit ed9be06)
Signed-off-by: Yin Huai <yhuai@databricks.com>
asfgit pushed a commit that referenced this pull request May 7, 2015
Thank nadavoosh point this out in #5590

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes #5877 from adrian-wang/jdbcrdd and squashes the following commits:

cc11900 [Daoyuan Wang] avoid NPE in jdbcrdd

(cherry picked from commit ed9be06)
Signed-off-by: Yin Huai <yhuai@databricks.com>
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 14, 2015
This pr convert java.sql.Date type into Int for JDBCRDD.

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes apache#5590 from adrian-wang/datebug and squashes the following commits:

f897b81 [Daoyuan Wang] add a test case
3c9184c [Daoyuan Wang] fix date type convertion in jdbcrdd
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
Thank nadavoosh point this out in apache#5590

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes apache#5877 from adrian-wang/jdbcrdd and squashes the following commits:

cc11900 [Daoyuan Wang] avoid NPE in jdbcrdd
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
Thank nadavoosh point this out in apache#5590

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes apache#5877 from adrian-wang/jdbcrdd and squashes the following commits:

cc11900 [Daoyuan Wang] avoid NPE in jdbcrdd
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
This pr convert java.sql.Date type into Int for JDBCRDD.

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes apache#5590 from adrian-wang/datebug and squashes the following commits:

f897b81 [Daoyuan Wang] add a test case
3c9184c [Daoyuan Wang] fix date type convertion in jdbcrdd
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
Thank nadavoosh point this out in apache#5590

Author: Daoyuan Wang <daoyuan.wang@intel.com>

Closes apache#5877 from adrian-wang/jdbcrdd and squashes the following commits:

cc11900 [Daoyuan Wang] avoid NPE in jdbcrdd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants