Fix pagination for GetMultipleKeys, introducing startAfterKey #38
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fm3
commented
May 4, 2023
| @@ -30,7 +30,7 @@ class FossilDBSuite extends FlatSpec with BeforeAndAfterEach with TestHelpers wi | |||
| private val testData3 = ByteString.copyFromUtf8("testData3") | |||
|
|
|||
| private val aKey = "aKey" | |||
| private val anotherKey = "anotherKey" | |||
| private val aNotherKey = "aNotherKey" | |||
There was a problem hiding this comment.
this change to make the tests more intuitive (keys are alphabetically sorted now)
| @@ -91,8 +93,6 @@ class VersionedKeyValueStore(underlying: RocksDBStore) { | |||
| def get(key: String, version: Option[Long] = None): Option[VersionedKeyValuePair[Array[Byte]]] = | |||
| scanVersionValuePairs(key, version).toStream.headOption | |||
|
|
|||
| def getUnderlying: RocksDBStore = underlying | |||
fm3
changed the title
6 tasks
|
@frcroth I know you are not familiar with this codebase yet, let’s have a talk about this in person (but feel free bo have a look already) |
frcroth
approved these changes
May 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes a bug where the pagination for
GetMultipleKeyscould lead to an endless loop if some keys are prefixes of others.This happened because the scanKeys method would scan to the supplied start key, which the client would set to the last key of the previous response. However, if that key is a prefix of another key, that other key might come earlier in the sorted list, leading the scan operation to end up up there.
This is fixed by adding the version separator
@to the start key before scanning to it. This ensures that prefix matches are not scanned to (since it can’t be a prefix if it includes the reserved character@).However, that made it impossible to supply a start key that is not actually present. Since clients previously supplied the common prefix of the keys as the very first start key during pagination (because it was non-optional, but the clients don’t know the first actual key), they will not get the same results anymore. So I’d rather mark this as a breaking change, and while I’m on it, unify the pagination logic with ListKeys (introducing an optional startAfterKey instead of the ambiguous start key).
So the
GetMultipleKeyscall now takes astartAfterKeyinstead of akeyfor pagination. The returned list will only start after this key.I adapted the tests and added some more to ensure and document the functionality of GetMultipleKeys