Skip to content

Commit

Permalink
[SPARK-10392] [SQL] Pyspark - Wrong DateType support on JDBC connection
Browse files Browse the repository at this point in the history
This PR addresses issue [SPARK-10392](https://issues.apache.org/jira/browse/SPARK-10392)
The problem is that for "start of epoch" date (01 Jan 1970) PySpark class DateType returns 0 instead of the `datetime.date` due to implementation of its return statement

Issue reproduction on master:
```
>>> from pyspark.sql.types import *
>>> a = DateType()
>>> a.fromInternal(0)
0
>>> a.fromInternal(1)
datetime.date(1970, 1, 2)
```

Author: 0x0FFF <programmerag@gmail.com>

Closes #8556 from 0x0FFF/SPARK-10392.
  • Loading branch information
0x0FFF authored and davies committed Sep 1, 2015
1 parent bf550a4 commit 00d9af5
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 2 deletions.
5 changes: 5 additions & 0 deletions python/pyspark/sql/tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,11 @@ def test_decimal_type(self):
t3 = DecimalType(8)
self.assertNotEqual(t2, t3)

# regression test for SPARK-10392
def test_datetype_equal_zero(self):
dt = DateType()
self.assertEqual(dt.fromInternal(0), datetime.date(1970, 1, 1))


class SQLTests(ReusedPySparkTestCase):

Expand Down
6 changes: 4 additions & 2 deletions python/pyspark/sql/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -168,10 +168,12 @@ def needConversion(self):
return True

def toInternal(self, d):
return d and d.toordinal() - self.EPOCH_ORDINAL
if d is not None:
return d.toordinal() - self.EPOCH_ORDINAL

def fromInternal(self, v):
return v and datetime.date.fromordinal(v + self.EPOCH_ORDINAL)
if v is not None:
return datetime.date.fromordinal(v + self.EPOCH_ORDINAL)


class TimestampType(AtomicType):
Expand Down

0 comments on commit 00d9af5

Please sign in to comment.