Very high I/O load

[kaber2]

I'm running multiple nodes for Ethereum, BSC, Heco, Fantom, ... as well as Avalanche. I've noticed Avalanche is causing excessively high I/O load compared to every other blockchain I'm running.

On this particular system, over a time window of 5 minutes, BSC has caused 130kb of disk writes, Fantom 60kb, Harmony 17MB while Avalanche has caused 23GB of disk writes, so more than a factor of 1000 more than the next blockchain. All nodes are fully synced up, the same picture is visible on all my other systems. The I/O load is attributed to avalanchego, not coreth.

Given the much higher transaction load on f.i. BSC, the Tx load itself doesn't appear to be the cause of this. I'm assuming it has something to do with write caching done by the other blockchains.

Do you have any hints what might be causing this, and ideally, how to get this down to a more reasonable level?

Thanks!

[kaber2]

Just another datapoint, I've tried to increase the caching settings to what I normally use for my other nodes (32GB of cache):

(updated diff formatting:)

--- a/eth/ethconfig/config.go
+++ b/eth/ethconfig/config.go
@@ -54,13 +54,13 @@ func NewDefaultConfig() Config {
                NetworkId:               1,
                LightPeers:              100,
                UltraLightFraction:      75,
+               DatabaseCache:           32768 * 50 / 100,
+               TrieCleanCache:          32768 * 10 / 100,
                TrieCleanCacheJournal:   "triecache",
                TrieCleanCacheRejournal: 60 * time.Minute,
+               TrieDirtyCache:          32768 * 30 / 100,
                TrieTimeout:             60 * time.Minute,
+               SnapshotCache:           32768 * 20 / 100,
                Miner:                   miner.Config{},
                TxPool:                  core.DefaultTxPoolConfig,
                RPCGasCap:               25000000,

But there's no noticeable difference, its still using roughly the same amount of I/O bandwidth and also doesn't seem to be using more memory for the cache.

[t-anyu]

I can also attest to this. The memory usage is extremely small, but IO are very high. Tried to alter db blockCacheSize, but didn't show any difference.

Would be very nice to have tuning parameter or a flag to increase memory usage and lower disk usage. To save some nvmes.

[heki-ru]

Confirming, latest avalanchego constantly doing 200-300 MB/s write read operations, causing high IOWAIT and disrupting work of other services. And it's not on the program start only, but during regular run

[StephenButtolph]

v1.7.7 should significantly reduce the DB IO used by avalanche. If the issue still seems to persist after upgrading to v1.7.7 please create a new issue.