MaxDegreeOfParallelism defaults for full pruning #5662
Conversation
MarekM25
requested review from
asdacap,
LukaszRozmej,
kamilchodola and
cbermudez97
|
Option 3 for me looks the best - but remember to take a ceiling value out of it. Better slightly more than slightly less. |
Yeah, I think about option 3 or 4. Yes as I described in the issues I am aware of the correct safeguards :) |
|
Just remember about one thing - if we will change defaults and someone decide to use the default, then they will experience quite enormous degradation of pruning time as soon as we will do that. From UX point of view changing for some users from current (maybe) 16 cores down to 3 may be a horrible experience (even if they will have better avg of block processing - ling term it will be the same outcome because it will be slightly worse for longer period of time vs significantly worse but for shorter period). |
| @@ -37,7 +37,7 @@ public interface IPruningConfig : IConfig | |||
| FullPruningTrigger FullPruningTrigger { get; set; } | |||
|
|
|||
| [ConfigItem( | |||
| Description = "'Full' pruning: Defines how many parallel tasks and potentially used threads can be created by full pruning. 0 - number of logical processors, 1 - full pruning will run on single thread. " + | |||
There was a problem hiding this comment.
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.paralleloptions.maxdegreeofparallelism?view=net-7.0 -1 in .NET means Environment.ProcessorCount
|
I like the 50% option. |
50% of real cores rather than HT cores? |
* other defaults for full pruning? * degreeOfParalleism * small refactor * fix build * Fixing BatchedTrieVistior * 25% of cores * Update Pruning config * add logger * fix
* Add missing peer types to limits (#5838) * Add missing peer types to limits * Add Reth * Flaky test * Use better processing timer for logging (#5843) * Use different processing timer * Use single queue * Revert to simpler * Add Gnosis Shanghai hard-fork timestamp (#5848) * Don't use DarkGray (#5849) * Bump to 1.20.0-rc * MaxDegreeOfParallelism defaults for full pruning (#5662) * other defaults for full pruning? * degreeOfParalleism * small refactor * fix build * Fixing BatchedTrieVistior * 25% of cores * Update Pruning config * add logger * fix * Perf/smoother peer discovery (#5846) * Continuously connect * Added another delay due to disconnect * Increase useless peer timeout * Simple rate limiter * Integrate with peer manager * Adjust some stats * Check all peer, and uses pending variable * Make logic clearer * Minor cleanup * Missed cancellation token * Cancel setup outgoing peer connection if throttled * Lockless ratelimiter * Addressing comment * Minor adjustments * Whitespace * Having trouble getting candidate at higher speed * Make test more consistent * Even more lenient (cherry picked from commit 6ab0a3a) * Update RocksDB package (#5883) (cherry picked from commit 4e2bf0f) * Update DotNetty feed (cherry picked from commit 3e330ad) * Update DotNetty package (cherry picked from commit 47a92ee) * add more gnosis bootnodes (#5910) (cherry picked from commit b7086a9) * Remove `v` of non-legacy tx signature from RPC response (#5927) * minor fixes * Update version to 1.20.0 * Bump to 1.20.1 * Restore `v` in tx signature for Geth compatibility (#5937) * Restore DotNetty feed (#5976) * Revert `nuget.config` * Update DotNetty package --------- Co-authored-by: Ben Adams <thundercat@illyriad.co.uk> Co-authored-by: Ruben Buniatyan <rubo@users.noreply.github.com> Co-authored-by: Kamil Chodoła <kamil@nethermind.io> Co-authored-by: Marek Moraczyński <marekm2504@gmail.com> Co-authored-by: Amirul Ashraf <asdacap@gmail.com> Co-authored-by: Marcin Sobczak <marcindsobczak@gmail.com>
Changes
We aim to provide default options that meet the needs of most people. 95% of our users want to have full pruning that ideally doesn't affect block processing. The current default
Pruning.FullPruningMaxDegreeOfParallelism = Environment.ProcessorCountisn't good as it will murder block processing and affects people's attestations. This PR doesn't suppose to be merged as it is but rather an open discussion about defaults.We have the following options, all of which seem to be better than the current default:
Pruning.FullPruningMaxDegreeOfParallelism = 2,Pruning.FullPruningMaxDegreeOfParallelism = 3Pruning.FullPruningMaxDegreeOfParallelism = 0but under the hood, it translates toEnvironment.ProcessorCount / 2Pruning.FullPruningMaxDegreeOfParallelism = 0but under the hood, it translates toEnvironment.ProcessorCount - 2Pruning.FullPruningMaxDegreeOfParallelism = 0but under the hood, it translates toEnvironment.ProcessorCount - 1Of course, we will add correct safeguards for Math.Max or Math.Min depends on the option, for example for
Pruning.FullPruningMaxDegreeOfParallelism = 3it could be Math.Min(Environment.ProcessorCount, 3)WDYT @asdacap @LukaszRozmej @kamilchodola @cbermudez97 ?
Types of changes
What types of changes does your code introduce?
Testing
Requires testing
If yes, did you write tests?
Notes on testing
Not now