-
Notifications
You must be signed in to change notification settings - Fork 22
CASSANALYTICS-102: Add TimeRangeFilter to filter out SSTables outside given time window #155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
df991d5 to
58d3337
Compare
yifan-c
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not see the validation that check the target table is using TWCS when the time range filter is configured. Should you add the check?
...lytics-common/src/main/java/org/apache/cassandra/spark/sparksql/filters/TimeRangeFilter.java
Outdated
Show resolved
Hide resolved
...lytics-common/src/main/java/org/apache/cassandra/spark/sparksql/filters/TimeRangeFilter.java
Outdated
Show resolved
Hide resolved
...lytics-common/src/main/java/org/apache/cassandra/spark/sparksql/filters/TimeRangeFilter.java
Outdated
Show resolved
Hide resolved
...lytics-common/src/main/java/org/apache/cassandra/spark/sparksql/filters/TimeRangeFilter.java
Outdated
Show resolved
Hide resolved
...cs-common/src/test/java/org/apache/cassandra/spark/sparksql/filters/TimeRangeFilterTest.java
Outdated
Show resolved
Hide resolved
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/utils/FilterUtils.java
Outdated
Show resolved
Hide resolved
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/utils/FilterUtils.java
Outdated
Show resolved
Hide resolved
cassandra-analytics-core/src/test/java/org/apache/cassandra/spark/data/LocalDataLayerTests.java
Outdated
Show resolved
Hide resolved
cassandra-analytics-core/src/test/java/org/apache/cassandra/spark/SSTableReaderTests.java
Outdated
Show resolved
Hide resolved
...n-tests/src/test/java/org/apache/cassandra/analytics/BulkReaderFilteringIntegrationTest.java
Outdated
Show resolved
Hide resolved
yifan-c
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One last comment
| public static final SSTableTimeRangeFilter EMPTY = new SSTableTimeRangeFilter(Range.closed(0L, 0L)) | ||
| { | ||
| @Override | ||
| public boolean overlaps(long startMicros, long endMicros) | ||
| { | ||
| return true; // accepts all SSTables | ||
| } | ||
| }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I originally suggested, the empty filter was with range Range.openClosed(0L, 0L). The filter does not overlap with any sstables, hence it is named EMPTY. It is to replace the usage of null.
The code here creates a filter that overlaps with all SSTables. Now I realize that this is probably the behavior you want. Could you rename the filter? It is not really EMPTY, but ALL. The code can be simplified to
| public static final SSTableTimeRangeFilter EMPTY = new SSTableTimeRangeFilter(Range.closed(0L, 0L)) | |
| { | |
| @Override | |
| public boolean overlaps(long startMicros, long endMicros) | |
| { | |
| return true; // accepts all SSTables | |
| } | |
| }; | |
| public static final SSTableTimeRangeFilter ALL = new SSTableTimeRangeFilter(Range.all()); |
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/data/DataLayer.java
Outdated
Show resolved
Hide resolved
88f2cfb to
f420c8a
Compare
No description provided.