client: don’t update validators produced counters when syncing #6762
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EpochManager::get_validator_info call querying information by block hash turns out to be quite expensive while node is synchronising. Normally, EpochManager maintains validator information up to the latest final block. With that, calling get_validator_info with a block hash usually needs to iterate over only a handful of blocks until it hits the latest final block and then can read the already aggregated data. However, while node is syncing, the aggregated data is not maintained. This means that getting validator info for a given block requires iteration over all blocks until start of the epoch. This is an expensive operation. EpochManager::get_validator_info is called each time stats line is generated. When synchronising this ends up taking a up all of the processing time to the point that node doesn’t respond to /status queries. Stop getting the validator information when node is synchronising to avoid this problem. The observable effect is that - near_validators_blocks_expected, - near_validators_blocks_produced, - near_validators_chunks_expected and - near_validators_chunks_produced metrics are not updated.
bowenwang1996
approved these changes
May 6, 2022
marcelo-gonzalez
pushed a commit
that referenced
this pull request
May 10, 2022
EpochManager::get_validator_info call querying information by block hash turns out to be quite expensive while node is synchronising. Normally, EpochManager maintains validator information up to the latest final block. With that, calling get_validator_info with a block hash usually needs to iterate over only a handful of blocks until it hits the latest final block and then can read the already aggregated data. However, while node is syncing, the aggregated data is not maintained. This means that getting validator info for a given block requires iteration over all blocks until start of the epoch. This is an expensive operation. EpochManager::get_validator_info is called each time stats line is generated. When synchronising this ends up taking a up all of the processing time to the point that node doesn’t respond to /status queries. Stop getting the validator information when node is synchronising to avoid this problem. The observable effect is that - near_validators_blocks_expected, - near_validators_blocks_produced, - near_validators_chunks_expected and - near_validators_chunks_produced metrics are not updated. Issue: #6763
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
EpochManager::get_validator_info call querying information by block
hash turns out to be quite expensive while node is synchronising.
Normally, EpochManager maintains validator information up to the
latest final block. With that, calling get_validator_info with
a block hash usually needs to iterate over only a handful of blocks
until it hits the latest final block and then can read the already
aggregated data.
However, while node is syncing, the aggregated data is not maintained.
This means that getting validator info for a given block requires
iteration over all blocks until start of the epoch. This is an
expensive operation.
EpochManager::get_validator_info is called each time stats line is
generated. When synchronising this ends up taking a up all of the
processing time to the point that node doesn’t respond to /status
queries.
Stop getting the validator information when node is synchronising to
avoid this problem. The observable effect is that
metrics are not updated.
Issue: #6763