New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC-135: PPD for timestamp is wrong when reader and writer timezones are different #87

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@prasanthj
Contributor

prasanthj commented Jan 27, 2017

When reader and writer timezones are different, PPD evaluation does not offset the timezone when reading the min and max values. This can result is wrong PPD evaluation and hence incorrect results.

Example:
Table written in US/Eastern timezone. All values in this table are "2007-08-01 00:00:00.0".
PPD disabled

hive> set hive.optimize.index.filter=false;
hive> select ORDER_DATE from ORDER_FACT_small where ORDER_DATE='2007-08-01 00:00:00.0' limit 1;
2007-08-01 00:00:00.0
OK

PPD enabled

set hive.optimize.index.filter=true;
select ORDER_DATE from ORDER_FACT_small where ORDER_DATE='2007-08-01 00:00:00.0' limit 1;
OK

No rows are returned when PPD is enabled (reader timezone is UTC)

@@ -19,6 +19,9 @@
import java.sql.Date;
import java.sql.Timestamp;
import java.util.Calendar;
import java.util.GregorianCalendar;
import java.util.TimeZone;

This comment has been minimized.

@prasanthj

prasanthj Jan 27, 2017

Contributor

These imports are not required. Will remove it in the next patch.

@prasanthj

prasanthj Jan 27, 2017

Contributor

These imports are not required. Will remove it in the next patch.

@omalley

This comment has been minimized.

Show comment
Hide comment
@omalley

omalley Feb 1, 2017

Contributor

Ok, I took a pass at this:

https://github.com/omalley/orc/tree/orc-135

Changes:

  • Block all use of PPD on timestamps from before ORC-135.
  • Block PPD on timestamp with numeric constraints
  • TimestampColumnStatistics.getMinimum and getMaximum return Timestamps in the local timezone.
  • Loading old TimestampColumnStatistics loads the min and max assuming local timezone.
  • Add utilities to SerializationUtils for conversion to and from utc.
  • Make BloomFilter.encoding an int so that we can detect future values.
  • Future BloomFilter encodings are not used.
  • Some more test cases
Contributor

omalley commented Feb 1, 2017

Ok, I took a pass at this:

https://github.com/omalley/orc/tree/orc-135

Changes:

  • Block all use of PPD on timestamps from before ORC-135.
  • Block PPD on timestamp with numeric constraints
  • TimestampColumnStatistics.getMinimum and getMaximum return Timestamps in the local timezone.
  • Loading old TimestampColumnStatistics loads the min and max assuming local timezone.
  • Add utilities to SerializationUtils for conversion to and from utc.
  • Make BloomFilter.encoding an int so that we can detect future values.
  • Future BloomFilter encodings are not used.
  • Some more test cases

@asfgit asfgit closed this in f2b8b79 Feb 7, 2017

asfgit pushed a commit that referenced this pull request Feb 7, 2017

ORC-135: PPD for timestamp is wrong when reader and writer timezones …
…are different

zones. (prasanthj and omalley)

Fixes #87
Fixes #90

Signed-off-by: Owen O'Malley <omalley@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment