Anvil is consuming high disk space ( more than ~ 25Gi in 1 hour)

[AnieeG]

Component

Anvil

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

anvil 0.2.0 (ed79650 2024-06-29T05:54:37.892564901Z)

What command(s) is the bug in?

anvil --chain-id 90000038 --block-time 1 --gas-limit 100000000 --code-size-limit 30000 --base-fee 1000000 --host 0.0.0.0 --port 8545 --accounts 20 --balance 2000000000000000000 --state /state/dump.json --state-interval 60 --transaction-block-keeper 500

Operating System

Linux

Describe the bug

With the recent anvil version the disk usage in anvil working dir is getting extremely high.
Here are some usage data when I ran du on the container's working dir.

4.0M    ./lib
31.9M   ./state
207.0M  ./usr
24.8G   ./home/foundry/.foundry/anvil/tmp/anvil-state-08-07-2024-22-10fvxvIm   

I am performing a load test which used to work with forge 0.2.0 (0a5b22f 2024-05-21T10:30:02.156279959Z)
but with the recent version upgrade the high disk usage is causing the pod to get evicted.
The load test sends a high volume of tx into the anvil node ( ~ 10-100 tx per 5 second ).

Is there any recent update which can be the cause of this high disk consumption? Is there any work around for this issue?

[grandizzy]

most probably related to the fact that since #8160 all blocks are dumped in json file

[AnieeG]

Is there a plan to optimize the consumption? I am not able to find the previous working version in the list of foundry images. Is there any suggested version I can use to avoid this issue?

[grandizzy]

do you need to keep the state dump during test load or could you just prune it periodically?

[AnieeG]

I need to keep the state, I tried to use --prune-history but getting the following error -
BlockOutOfRangeError: block height is 3491 but requested was 3492

[grandizzy]

you could probably try with this version https://github.com/foundry-rs/foundry/releases/tag/nightly-5ac78a9cd4b94dc53d1fe5e0f42372b28b5a7559

[grandizzy]

also wonder if setting up something like a logrotate job to compress file if certain size exceeded would work

[kalverra]

also wonder if setting up something like a logrotate job to compress file if certain size exceeded would work

Would this compression disallow reading that block data?

[AnieeG]

I have usedanvil 0.2.0 (5ac78a9 2024-06-02T00:20:14.684725967Z)and still see this

Is there a way to find the image related to this version - forge 0.2.0 (0a5b22f 2024-05-21T10:30:02.156279959Z)

[grandizzy]

also wonder if setting up something like a logrotate job to compress file if certain size exceeded would work

Would this compression disallow reading that block data?

yes, they would need to be uncompressed in order to read it

[grandizzy]

I have usedanvil 0.2.0 (5ac78a9 2024-06-02T00:20:14.684725967Z)and still see this

Is there a way to find the image related to this version - forge 0.2.0 (0a5b22f 2024-05-21T10:30:02.156279959Z)

hm, I thought is because the /state/dump.json size, but looks it is rather the anvil state cache. Can you pls confirm the size of /state/dump.json ?

[grandizzy]

the max_on_disk_limit

foundry/crates/anvil/src/eth/backend/mem/storage.rs

Lines 58 to 61 in 82ff8ee

	/// maximum amount of states we keep on disk
	///
	/// Limiting the states will prevent disk blow up, especially in interval mining mode
	max_on_disk_limit: usize,

is set to 3600

foundry/crates/anvil/src/eth/backend/mem/storage.rs

Lines 45 to 46 in 82ff8ee

	// 1hr of up-time at lowest 1s interval
	const MAX_ON_DISK_HISTORY_LIMIT: usize = 3_600;

I think would make sense to have it configurable for handling scenarios like in this ticket, @mattsse wdyt?

[mattsse]

definitely, opening an issue for this: #8399

[AnieeG]

I have usedanvil 0.2.0 (5ac78a9 2024-06-02T00:20:14.684725967Z)and still see this
Is there a way to find the image related to this version - forge 0.2.0 (0a5b22f 2024-05-21T10:30:02.156279959Z)

hm, I thought is because the /state/dump.json size, but looks it is rather the anvil state cache. Can you pls confirm the size of /state/dump.json ?

The /state/dump.json is quite alright. In the screenshot I posted in description of this issue you will find 31.9M ./state whereas the /home cache has grown huge - 24.8G ./home/foundry/.foundry/anvil/tmp/anvil-state-08-07-2024-22-10fvxvIm

[AnieeG]

the max_on_disk_limit

foundry/crates/anvil/src/eth/backend/mem/storage.rs

Lines 58 to 61 in 82ff8ee

	/// maximum amount of states we keep on disk
	///
	/// Limiting the states will prevent disk blow up, especially in interval mining mode
	max_on_disk_limit: usize,

is set to 3600

foundry/crates/anvil/src/eth/backend/mem/storage.rs

Lines 45 to 46 in 82ff8ee

	// 1hr of up-time at lowest 1s interval
	const MAX_ON_DISK_HISTORY_LIMIT: usize = 3_600;

I think would make sense to have it configurable for handling scenarios like in this ticket, @mattsse wdyt?

Could you please also add a documentation for this? If the MAX_ON_DISK_HISTORY_LIMIT is over will I encounter similar error (BlockOutOfRangeError: block height is 3491 but requested was 3492) as I was getting for --prune-history option?

[grandizzy]

MAX_ON_DISK_HISTORY_LIMIT

the MAX_ON_DISK_HISTORY_LIMIT already purge the oldest state cached on disk, which should also fix the issue you see with--prune-history

[AnieeG]

@grandizzy we increased the available disc space, but we keep on getting error like this even without --prune-history enabled - BlockOutOfRangeError: block height is 11112 but requested was 11113

This is blocking us significantly. Would really appreciate any suggestion to resolve this -

command used -

anvil --chain-id 90000050 --block-time 1 --gas-limit 100000000 --code-size-limit 30000 --base-fee 1000000 --host 0.0.0.0 --port 8545 --accounts 20 --balance 2000000000000000000 --state /state/dump.json --state-interval 60 --transaction-block-keeper 500

[grandizzy]

@AnieeG there's a PR in works to address this #8412 will be avail soon