This repository has been archived by the owner on Feb 21, 2024. It is now read-only.
allow translate log entry buffer to grow #1787
Merged
travisturner
merged 3 commits into
FeatureBaseDB:master
from
travisturner:translate-read-buffer-increase
Dec 18, 2018
Merged
allow translate log entry buffer to grow #1787
travisturner
merged 3 commits into
FeatureBaseDB:master
from
travisturner:translate-read-buffer-increase
Dec 18, 2018
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the case where a translate log entry contained many key/id pairs, it was possible for the read buffer (which was allocated at 65536 bytes) to fail to handle it. This happened when the serialized LogEntry was larger than 65536 bytes. This PR adds logic which returns a custom error called ErrTranslateReadTargetUndersized notifying the reader to reallocate a larger read buffer and try the read again. TODO: - [ ] Add a max buffer size check to prevent this from doubling the buffer size with no limit. - [ ] Add tests.
yuce
previously approved these changes
Dec 18, 2018
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR fixes the original problem in #1784 I think merging this PR and tackling the other problem in that ticket after the release is reasonable.
ok, i'll address the TODOs and send it back to you |
travisturner
force-pushed
the
translate-read-buffer-increase
branch
from
December 18, 2018 18:48
6b93dc0
to
358c32a
Compare
@yuce I completed the TODOs on this PR. |
travisturner
changed the title
WIP: allow translate log entry buffer to grow
allow translate log entry buffer to grow
Dec 18, 2018
Tests run OK, testing it with a sample cluster. |
The original problem remains resolved. Running the second node after import and immediately querying it is still problematic. Here are Pilosa logs of two outcomes in case it helps to diagnose the problem later:
|
yuce
approved these changes
Dec 18, 2018
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
In the case where a translate log entry contained
many key/id pairs, it was possible for the read
buffer (which was allocated at 65536 bytes) to
fail to handle it. This happened when the serialized
LogEntry
was larger than 65536 bytes.This PR adds logic which returns a custom error called
ErrTranslateReadTargetUndersized
notifying the readerto reallocate a larger read buffer and try the read
again.
TODO:
buffer size with no limit.
Fixes #1784
Pull request checklist
Code review checklist
This is the checklist that the reviewer will follow while reviewing your pull request. You do not need to do anything with this checklist, but be aware of what the reviewer will be looking for.