feat(projectconfig): Decompress redis value [INGEST-1505 INGEST-1506] #1345
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjbayer
commented
Jul 21, 2022
| fn parse_redis_response(raw_response: &[u8]) -> Result<ProjectState, RedisProjectError> { | ||
| let decoded = metric!(timer(RelayTimers::ProjectStateDecompression), { | ||
| zstd::decode_all(raw_response) | ||
| }); |
There was a problem hiding this comment.
Instead of checking for zstandard's magic number explicitly, I just run decompression here, assuming that the library already efficiently checks if the given blob is in zstandard format. We can use the logged metric to verify this.
There was a problem hiding this comment.
that makes sense. I wonder whether zstd::decode_all will allocate a vector in case the payload is not zstd. but i think either way it'll work
untitaker
reviewed
Jul 21, 2022
| }); | ||
|
|
||
| let raw_response: Cow<[u8]> = match decoded { | ||
| Ok(decoded) => decoded.into(), |
There was a problem hiding this comment.
can't you just borrow here to avoid the Cow?
untitaker
approved these changes
Jul 21, 2022
This was referenced Jul 21, 2022
jjbayer
added a commit
that referenced
this pull request
Jul 25, 2022
Record more metrics for the decompression of project config entries in the redis cache (see #1345).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With transaction metrics and dynamic sampling config, the size of the project config values written to redis become a problem at scale. Compression should mitigate this problem.