prefetch: claim.sweat::claim #11094

nagisa · 2024-04-17T08:04:47Z

No description provided.

codecov · 2024-04-17T09:13:31Z

Codecov Report

Attention: Patch coverage is 24.80000% with 94 lines in your changes are missing coverage. Please review.

Project coverage is 71.05%. Comparing base (4d506a7) to head (afeadaa).
Report is 16 commits behind head on master.

Files	Patch %	Lines
runtime/runtime/src/prefetch.rs	4.76%	80 Missing ⚠️
core/store/src/trie/prefetching_trie_storage.rs	51.72%	8 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11094      +/-   ##
==========================================
- Coverage   71.14%   71.05%   -0.09%     
==========================================
  Files         761      764       +3     
  Lines      152857   153225     +368     
  Branches   152857   153225     +368     
==========================================
+ Hits       108748   108881     +133     
- Misses      39672    39902     +230     
- Partials     4437     4442       +5

Flag	Coverage Δ
backward-compatibility	`0.24% <0.00%> (-0.01%)`	⬇️
db-migration	`0.24% <0.00%> (-0.01%)`	⬇️
genesis-check	`1.43% <9.60%> (+<0.01%)`	⬆️
integration-tests	`36.79% <16.80%> (-0.05%)`	⬇️
linux	`69.47% <24.80%> (-0.15%)`	⬇️
linux-nightly	`70.55% <24.80%> (-0.09%)`	⬇️
macos	`54.23% <21.60%> (+1.60%)`	⬆️
pytests	`1.66% <9.60%> (+<0.01%)`	⬆️
sanity-checks	`1.45% <9.60%> (+<0.01%)`	⬆️
unittests	`66.75% <24.80%> (-0.05%)`	⬇️
upgradability	`0.29% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

So that we can see what's failing?

nagisa · 2024-04-17T13:40:45Z

Blockchain utilization without and with this PR
Apply latency without and with this PR

akashin · 2024-04-17T14:38:48Z

runtime/runtime/src/prefetch.rs

+                return;
+            };
+
+            for (dt, idx) in account_record.accruals {


Idea: if we think there could be especially large number of accruals in some of the records, it might make sense to limit the work in this thread to only prefetch the first K of them.

How do we know what limit to use? But also, given that this code is going to be dead in 3 weeks...

mooori · 2024-04-17T15:50:59Z

runtime/runtime/src/prefetch.rs

+            // Just read this directly for now since this is temporary anyway
+            let prefetcher_storage = prefetch_api.make_storage();
+            let trie = Trie::new(prefetcher_storage, trie_root, None);
+            let Ok(Some(account_record)) = trie.get(&trie_key.to_vec()) else {


The trie is based on TriePrefetchingStorage, which should ensure the data is not retrieved simultaneously by multiple threads

mooori · 2024-04-17T15:57:38Z

runtime/runtime/src/prefetch.rs

+                let Ok(()) = 1u8.serialize(&mut accruals_key) else { continue };
+                let Ok(()) = dt.serialize(&mut accruals_key) else { continue };
+                let accruals_key = sha2::Sha256::digest(&accruals_key).to_vec();
+                let _ = prefetch_api.prefetch_trie_key(


Going through prefetch_api.prefetch_trie_key puts this request at the end of the prefetcher's FIFO work queue.

Bypassing the queue might be avoided by using trie.get as above, to have the data available earlier. However bypassing the queue and prefetcher API like that for many keys is probably risky.

mooori · 2024-04-17T15:58:14Z

runtime/runtime/src/prefetch.rs

+                let Ok(()) = 2u8.serialize(&mut amount_key) else { continue };
+                let Ok(()) = dt.serialize(&mut amount_key) else { continue };
+                amount_key.extend(&idx.to_le_bytes()); // index into Vector
+                let _ = prefetch_api.prefetch_trie_key(


nagisa force-pushed the claim-sweat-claim-prefetch branch from 8a9786c to 527dcb6 Compare April 17, 2024 08:14

prefetch: claim.sweat::claim

5422c2c

nagisa force-pushed the claim-sweat-claim-prefetch branch from 527dcb6 to 5422c2c Compare April 17, 2024 08:46

nagisa force-pushed the claim-sweat-claim-prefetch branch from 2e2da38 to 71c26cc Compare April 17, 2024 10:32

tracing: add a prefetch failure log

0baca04

So that we can see what's failing?

nagisa force-pushed the claim-sweat-claim-prefetch branch from 71c26cc to 0baca04 Compare April 17, 2024 10:33

fix config filtering

0c23b2a

nagisa force-pushed the claim-sweat-claim-prefetch branch from b01c5b5 to 0c23b2a Compare April 17, 2024 11:49

fix accountrecord schema

018376f

nagisa marked this pull request as ready for review April 17, 2024 12:42

nagisa requested a review from a team as a code owner April 17, 2024 12:42

nagisa requested a review from wacban April 17, 2024 12:42

nagisa marked this pull request as draft April 17, 2024 12:45

nagisa requested review from akashin and bowenwang1996 and removed request for wacban April 17, 2024 12:45

nagisa marked this pull request as ready for review April 17, 2024 12:54

fix clippy

e62cd7e

nagisa force-pushed the claim-sweat-claim-prefetch branch from 5029b43 to e62cd7e Compare April 17, 2024 12:54

clear out sender field to make it clear it doesn't matter

afeadaa

nagisa requested a review from mooori April 17, 2024 14:08

bowenwang1996 approved these changes Apr 17, 2024

View reviewed changes

akashin reviewed Apr 17, 2024

View reviewed changes

akashin approved these changes Apr 17, 2024

View reviewed changes

nagisa added this pull request to the merge queue Apr 17, 2024

Merged via the queue into near:master with commit 0efb5f7 Apr 17, 2024
26 of 29 checks passed

nagisa deleted the claim-sweat-claim-prefetch branch April 17, 2024 15:13

VanBarbascu pushed a commit that referenced this pull request Apr 17, 2024

prefetch: claim.sweat::claim (#11094)

9ba8874

VanBarbascu mentioned this pull request Apr 17, 2024

Blockers and Wishlist for the 1.39.1 release #11015

Open

mooori reviewed Apr 17, 2024

View reviewed changes

mooori mentioned this pull request Apr 18, 2024

Introduce sweat.claim as a Locust workloads #11107

Closed

nagisa mentioned this pull request Apr 19, 2024

Prefetch storage for claim.sweat and earn.kaiching #10987

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prefetch: claim.sweat::claim #11094

prefetch: claim.sweat::claim #11094

nagisa commented Apr 17, 2024

codecov bot commented Apr 17, 2024 •

edited

Loading

nagisa commented Apr 17, 2024

akashin Apr 17, 2024

nagisa Apr 17, 2024

mooori Apr 17, 2024

mooori Apr 17, 2024

mooori Apr 17, 2024

prefetch: claim.sweat::claim #11094

prefetch: claim.sweat::claim #11094

Conversation

nagisa commented Apr 17, 2024

codecov bot commented Apr 17, 2024 • edited Loading

Codecov Report

nagisa commented Apr 17, 2024

akashin Apr 17, 2024

Choose a reason for hiding this comment

nagisa Apr 17, 2024

Choose a reason for hiding this comment

mooori Apr 17, 2024

Choose a reason for hiding this comment

mooori Apr 17, 2024

Choose a reason for hiding this comment

mooori Apr 17, 2024

Choose a reason for hiding this comment

codecov bot commented Apr 17, 2024 •

edited

Loading