Add retry to tolerate the offload index file read failure #12452
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
--- *Motivation* We met the ReadLedgerMetadata exception when reading the index file. The index file only read once, so it may not read all the data from the stream and cause the metadata read failed. We need to ensure the all data is read from the stream or the stream is end. When the stream is end, we will receive the EOF exception, so we need to use `readFully` not `read`. Add the retry logic to tolerate the failure cause by the network. Because the stream is from the HTTP, so it's may break on some case. Add a small retry to avoid it to backoff by the dispatcher. *Modifications* - Use `readFully` to replace the `read` method - Add a small retry for handling the index block build
zymap
added
type/bug
zymap
requested review from
merlimat,
lhotari,
sijie,
hangc0276 and
codelipenghui
hangc0276
approved these changes
Oct 21, 2021
zymap
commented
Oct 21, 2021
| OffloadIndexBlock index; | ||
| try (InputStream payLoadStream = blob.getPayload().openStream()) { | ||
| index = (OffloadIndexBlock) indexBuilder.fromStream(payLoadStream); | ||
| int retryCount = 3; |
There was a problem hiding this comment.
This retry is for a quick retry, most of the time it shouldn't trigger. We just don't want to exit to wait for the next backoff.
codelipenghui
approved these changes
Oct 22, 2021
zymap
added a commit
that referenced
this pull request
Oct 22, 2021
* Add retry to tolerate the offload index file read failure --- *Motivation* We met the ReadLedgerMetadata exception when reading the index file. The index file only read once, so it may not read all the data from the stream and cause the metadata read failed. We need to ensure the all data is read from the stream or the stream is end. When the stream is end, we will receive the EOF exception, so we need to use `readFully` not `read`. Add the retry logic to tolerate the failure cause by the network. Because the stream is from the HTTP, so it's may break on some case. Add a small retry to avoid it to backoff by the dispatcher. *Modifications* - Use `readFully` to replace the `read` method - Add a small retry for handling the index block build * Add comments and enrich log (cherry picked from commit 33bcc17)
eolivelli
pushed a commit
to eolivelli/pulsar
that referenced
this pull request
Nov 29, 2021
* Add retry to tolerate the offload index file read failure --- *Motivation* We met the ReadLedgerMetadata exception when reading the index file. The index file only read once, so it may not read all the data from the stream and cause the metadata read failed. We need to ensure the all data is read from the stream or the stream is end. When the stream is end, we will receive the EOF exception, so we need to use `readFully` not `read`. Add the retry logic to tolerate the failure cause by the network. Because the stream is from the HTTP, so it's may break on some case. Add a small retry to avoid it to backoff by the dispatcher. *Modifications* - Use `readFully` to replace the `read` method - Add a small retry for handling the index block build * Add comments and enrich log
codelipenghui
pushed a commit
that referenced
this pull request
Dec 20, 2021
* Add retry to tolerate the offload index file read failure --- *Motivation* We met the ReadLedgerMetadata exception when reading the index file. The index file only read once, so it may not read all the data from the stream and cause the metadata read failed. We need to ensure the all data is read from the stream or the stream is end. When the stream is end, we will receive the EOF exception, so we need to use `readFully` not `read`. Add the retry logic to tolerate the failure cause by the network. Because the stream is from the HTTP, so it's may break on some case. Add a small retry to avoid it to backoff by the dispatcher. *Modifications* - Use `readFully` to replace the `read` method - Add a small retry for handling the index block build * Add comments and enrich log (cherry picked from commit 33bcc17)
lhotari
pushed a commit
to datastax/pulsar
that referenced
this pull request
Apr 11, 2022
* Add retry to tolerate the offload index file read failure --- *Motivation* We met the ReadLedgerMetadata exception when reading the index file. The index file only read once, so it may not read all the data from the stream and cause the metadata read failed. We need to ensure the all data is read from the stream or the stream is end. When the stream is end, we will receive the EOF exception, so we need to use `readFully` not `read`. Add the retry logic to tolerate the failure cause by the network. Because the stream is from the HTTP, so it's may break on some case. Add a small retry to avoid it to backoff by the dispatcher. *Modifications* - Use `readFully` to replace the `read` method - Add a small retry for handling the index block build * Add comments and enrich log (cherry picked from commit 33bcc17)
lhotari
pushed a commit
to datastax/pulsar
that referenced
this pull request
Apr 11, 2022
* Add retry to tolerate the offload index file read failure --- *Motivation* We met the ReadLedgerMetadata exception when reading the index file. The index file only read once, so it may not read all the data from the stream and cause the metadata read failed. We need to ensure the all data is read from the stream or the stream is end. When the stream is end, we will receive the EOF exception, so we need to use `readFully` not `read`. Add the retry logic to tolerate the failure cause by the network. Because the stream is from the HTTP, so it's may break on some case. Add a small retry to avoid it to backoff by the dispatcher. *Modifications* - Use `readFully` to replace the `read` method - Add a small retry for handling the index block build * Add comments and enrich log (cherry picked from commit 33bcc17)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
We met the ReadLedgerMetadata exception when reading the index
file. The index file only read once, so it may not read all the
data from the stream and cause the metadata read failed. We need
to ensure the all data is read from the stream or the stream is
end. When the stream is end, we will receive the EOF exception,
so we need to use
readFullynotread.Add the retry logic to tolerate the failure cause by the network.
Because the stream is from the HTTP, so it's may break on some
case. Add a small retry to avoid it to backoff by the dispatcher.
Modifications
readFullyto replace thereadmethodVerifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
doc-required(If you need help on updating docs, create a doc issue)
no-need-doc(Please explain why)
doc(If this PR contains doc changes)