PHOENIX-6211 Paged scan filters #973
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kadirozde
requested review from
ChinmaySKulkarni,
gjacoby126 and
abhishek-chouhan
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
gjacoby126
reviewed
Dec 14, 2020
| @@ -1343,6 +1346,36 @@ public int prepareIndexMutations(Put put, Delete del, Map<byte[], List<Mutation> | |||
| return indexMutations.size(); | |||
| } | |||
|
|
|||
| static boolean adjustScanFilter(Scan scan) { | |||
| // For rebuilds we use count (*) as query for regular tables which ends up setting the FKOF on scan | |||
There was a problem hiding this comment.
nit: please spell out FirstKeyOnlyFilter. Took me a minute to figure it out. :-)
| @@ -1406,6 +1407,7 @@ public void testWithVariousSQLsForMultipleViews() throws Exception { | |||
| } | |||
| } | |||
|
|
|||
| @Ignore("Fails with StaleRegionBoundaryCacheException. Mutations on a SCN connection could be the reason") | |||
There was a problem hiding this comment.
@jpisaac what are the implications of ignoring these tests?
There was a problem hiding this comment.
@gjacoby126 @kadirozde I have a fix for these test failures. It was failing in BaseScannerRegionObserver.preScannerOpen by throwing throwIfScanOutOfRegion when deletion of LocalIndexes are involved. All the ViewTTLIT tests are passing after the fix on my local machine
There was a problem hiding this comment.
@jpisaac - are the test fixes already in so that @kadirozde can rebase on them, or still just local to you?
There was a problem hiding this comment.
0001-Fixes-for-local-index-scans-and-additional-tests.patch.txt
@kadirozde @gjacoby126 I have attached the changes for the local index changes and some additional tests and added some comments on the PhoenixTTLRegionObserver
There was a problem hiding this comment.
@jpisaac, Thank you very much. I will apply your patch soon.
| if (!(scan.getFilter() instanceof PagedFilter)) { | ||
| byte[] pageSizeMsBytes = scan.getAttribute(BaseScannerRegionObserver.SERVER_PAGE_SIZE_MS); | ||
| if (pageSizeMsBytes != null) { | ||
| scan.setFilter(new PagedFilter(scan.getFilter(), Bytes.toLong(pageSizeMsBytes)/2)); |
There was a problem hiding this comment.
Why divided by 2? To allow for round trip time back to client? Good to have a comment.
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GlobalIndexRegionScanner.java
Show resolved
Hide resolved
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/GlobalIndexRegionScanner.java
Outdated
Show resolved
Hide resolved
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/PhoenixTTLRegionObserver.java
Show resolved
Hide resolved
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/PhoenixTTLRegionObserver.java
Outdated
Show resolved
Hide resolved
| @@ -26,6 +27,9 @@ | |||
| import org.apache.hadoop.hbase.util.Bytes; | |||
| import org.apache.phoenix.hbase.index.util.GenericKeyValueBuilder; | |||
| import org.apache.phoenix.util.PhoenixKeyValueUtil; | |||
| import org.apache.phoenix.util.ScanUtil; | |||
There was a problem hiding this comment.
nit: seems like several unnecessary imports?
|
💔 -1 overall
This message was automatically generated. |
gjacoby126
requested changes
Dec 18, 2020
| } | ||
|
|
||
| private boolean next(List<Cell> results, boolean raw) throws IOException { | ||
| boolean hasMore = raw ? delegate.nextRaw(results) : delegate.next(results); |
There was a problem hiding this comment.
if we throw an exception at either line 47 or line 56, do we clean up our state properly?
There was a problem hiding this comment.
I will add exception handling here.
phoenix-core/src/main/java/org/apache/phoenix/iterate/OffsetResultIterator.java
Show resolved
Hide resolved
phoenix-core/src/main/java/org/apache/phoenix/iterate/OrderedResultIterator.java
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
gjacoby126
reviewed
Jan 7, 2021
phoenix-core/src/it/java/org/apache/phoenix/end2end/ViewTTLIT.java
Outdated
Show resolved
Hide resolved
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/PagedRegionScanner.java
Show resolved
Hide resolved
|
@kadirozde - other than above comments lgtm. |
|
💔 -1 overall
This message was automatically generated. |
gjacoby126
approved these changes
Jan 7, 2021
There was a problem hiding this comment.
+1, thanks @kadirozde . Looks like you just need to rebase GlobalIndexChecker.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The design doc is at https://docs.google.com/document/d/1Vt28i9JLQPG3lAnbW3RcO7fEUG4KT5AhyydLMiau59k/edit?usp=sharing