Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unload obsolete memtries post block processing #11657

Merged
merged 3 commits into from
Jun 25, 2024
Merged

Conversation

staffik
Copy link
Contributor

@staffik staffik commented Jun 24, 2024

Currently we remove obsolete memtries only when we start state sync for a new shard: https://github.com/near/nearcore/blob/master/chain/client/src/client.rs#L2507-L2519
However, it does not unload obsolete memtries in all cases.
For example, if we tracked shards [0, 1, 2], and then we track shards [0, 1] only, shard 2 won't be unloaded, because we did not start a new state sync.
Although it is not critical, trie for shard 2 could occupy memory forever.

This PR calls the routine to unload obsolete memtries as a block post-processing step too.

@staffik staffik requested a review from a team as a code owner June 24, 2024 13:57
@staffik staffik requested a review from Longarithm June 24, 2024 13:57
Copy link

codecov bot commented Jun 24, 2024

Codecov Report

Attention: Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.

Project coverage is 71.61%. Comparing base (74ac5fe) to head (f58c2a1).
Report is 3 commits behind head on master.

Files Patch % Lines
chain/chain/src/chain.rs 91.30% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11657      +/-   ##
==========================================
- Coverage   71.65%   71.61%   -0.05%     
==========================================
  Files         787      787              
  Lines      161063   161104      +41     
  Branches   161063   161104      +41     
==========================================
- Hits       115413   115373      -40     
- Misses      40597    40684      +87     
+ Partials     5053     5047       -6     
Flag Coverage Δ
backward-compatibility 0.23% <0.00%> (-0.01%) ⬇️
db-migration 0.23% <0.00%> (-0.01%) ⬇️
genesis-check 1.36% <0.00%> (-0.01%) ⬇️
integration-tests 37.80% <91.30%> (+0.06%) ⬆️
linux 69.00% <91.30%> (-0.04%) ⬇️
linux-nightly 71.10% <91.30%> (-0.07%) ⬇️
macos 52.61% <91.30%> (+0.04%) ⬆️
pytests 1.59% <0.00%> (-0.01%) ⬇️
sanity-checks 1.39% <0.00%> (-0.01%) ⬇️
unittests 66.22% <91.30%> (-0.03%) ⬇️
upgradability 0.28% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Longarithm
Copy link
Member

That sounds right. However, still looks hacky that we are doing this check for every single block, while the shard "status" changes only once per epoch.

I have a feeling that we can call retain_mem_tries only if epoch_manager.is_next_block_epoch_start(parent_hash) is true, and remove retain_mem_tries from run_catchup because it doesn't fit there. Would you like to try it?

Also nit - if the function is called retain_mem_tries, the comment should also explain why retaining happens. The fact that it mentions GC (the opposite event) is confusing.

@staffik staffik added this pull request to the merge queue Jun 25, 2024
Merged via the queue into master with commit 524dc65 Jun 25, 2024
29 of 30 checks passed
@staffik staffik deleted the unload-memtrie-postblock branch June 25, 2024 08:53
staffik added a commit that referenced this pull request Jun 25, 2024
Currently we remove obsolete memtries only when we start state sync for
a new shard:
https://github.com/near/nearcore/blob/master/chain/client/src/client.rs#L2507-L2519
However, it does not unload obsolete memtries in all cases.
For example, if we tracked shards [0, 1, 2], and then we track shards
[0, 1] only, shard 2 won't be unloaded, because we did not start a new
state sync.
Although it is not critical, trie for shard 2 could occupy memory
forever.

This PR calls the routine to unload obsolete memtries as a block
post-processing step too.
@staffik staffik added the A-stateless-validation Area: stateless validation label Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-stateless-validation Area: stateless validation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants