ADR: HDFS Raster Layers #1582
Merged
ADR: HDFS Raster Layers #1582
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
||
| In a single value query we are given an instance of `K` and we must produce a corresponding `V` or an error. The first step is to locate the `MapFile` which potentially contains `(K, V)` record. Because the layer records are indexed by their SFC index we map `K` to `i: Long` and determine which file contains potential match by examining the file listing and finding the file with maximum starting index that is less than equal `i`. At this point the `MapFile` must be opened and queried for the key. | ||
|
|
||
| Because both listing all the files in the layer directory and opening a single MapFile we cache these steps. The file listing needs to be done only once per `HadoopValueReader` instance and we maintain an LRU cache of open `MapFile`s. Because SFC preserves some spatial locality of the records, geographically close records are likely to be close in SFC index, and we expect key/value queries to be geographically grouped, for instance requests from a map viewer, the `MapFile` LRU cache can be expected to have very high hit-rate. |
There was a problem hiding this comment.
Because both listing all the files in the layer directory and opening a single MapFile we cache these steps. I feel like this is missing some words.
|
+1 after changes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This ADR describes current implementation of the HDFS layers and key design decisions that were considered. This is intended to as both documentation and starting off point for further improvements.