Ideal System Resources

[xjmzx]

Just wondered what resources kibo needs most, and any potential bottlenecks when parsing the data.
I see that similar to IBD of chain data, there's more data to parse in more recent blocks.

My system seems to utilize about 13-14GB of memory.
Occassionaly CPU spikes, but consistently is around 10-25% using all cores (Ryzen 9 6900HX)
Disk I/O seems quite high utilization at 1.5-2GiB/s Read and1-2.5MiB/s Write.

So far, so good! Thanks for opening up a discussion area.

[nym21]

It needs in order of importance:

• a good amount of RAM-
• fast CPU
• fast IO

I'm curious how much ram it will use in the end, if you could tell then it would be really helpful as the number on Mac OS isn't the real one.

Basically kibo computes the full state of the chain for each block to compute all of the data which is quite a lot of resource and does it, in chunks of blocks in 1 month before flushing (exporting) everything to disk and starting the next chunk. I have plans to make this window (of 1 month) variable to be possible to use the program on machines with less RAM (smaller window, will be slower though) or more RAM (bigger window, to finish faster).
As for the disk io, it's high because once again we keep the full state of the chain at any time, which has to be saved on disk in databases otherwise you'd need a lot more RAM to keep it all in memory and they are read when needed (they're split in chunks also, to limit their growth).

Hope this answers your questions

[xjmzx]

Thanks for that. Explains all I was wanting to know. I'll post some updated stats when it gets towards the current time.

[xjmzx]

LANG=C free -k

               total        used        free      shared  buff/cache   available
Mem:        30528004    19221240      595148      231208    11391304    11306764
Swap:        8388604           0     8388604

free

               total        used        free      shared  buff/cache   available
Mem:        30528004    19279392      504768      231208    11424000    11248612
Swap:        8388604           0     8388604

Running on a standard desktop box. I noticed SWAP was full so switched it off and it put that 8G back into active memory.
Processing 2020-04-18 (height: 626465)...

[xjmzx]

[xjmzx]

Processing one month blocks seems to work well on my system. I was concerned that the parser would run into trouble after leaving it for a few days, but it seems okay.

If it happens to get interrupted, what are the implications?
Linux systemd service, or LaunchAgent on mac, would be helpful for server setups?

[nym21]

Thank you for the feedback ! It's more than I assumed, Mac OS really does some weird stuff with the RAM (my top usage in the monitor was 13 GB I think but was it true ?)

I did my best to avoid memory leaks so leaving it running should be fine (I have it running for days and days without issue)

As for the interruption of the parser, it's quite fragile (which I should've probably explain in the README).
So the program supports quitting with CTRL+C (SIGINT) and kill X (SIGTERM) both will be on standby if the program is in the process of exporting and then stop it or stop it directly if it's not.
Closing a terminal isn't supported for example, or shutting down the OS does first kill but if it takes too long while force kill, if I remember correctly, which also isn't supported.
When exporting we save to full state of the chain in databases if you stop in the middle of it it, the databases won't all be in sync which breaks the program as we only write changes and not the actual full state (for efficiency) and thus there is no easy way to go back.

As for the server, you can stop it whenever you want so you can definitely create a service that runs it on startup

If you have any other questions feel free to ask them !

[xjmzx]

I'm also running bitcoind (obviously) but also electrs, btcrpc, and a few other services, so may appear on the high side.

Question about the parser running. Do you mean that if the parser is interupted with ctl+c, it would need to start from the beginning of the chain again?

[nym21]

No don't worry, ctrl+c the parser or using kill PARSER_PID in the terminal should be completely safe !
But for example closing the terminal window where the parser is running or using kill -9 PARSER_PID (force kill) aren't safe and might corrupt the databases if the timing isn't right

[xjmzx]

Hey, just small update that my system finally completed the parser and runs fine on my desktop system. It took over a week for the parser to do its job (I suspected 16G memory might be a minimum. I logged the memory, disk and CPU periodically (manually) you can see the stats if any use:
https://gist.github.com/xjmzx/1fc69dbabd0bdaa6d24e799aae91d6fc

Good news is that I can run the server. After initial sync it uses about 8G memory on my system, and reads ~ 600MiB/s writes 40MiB/s at rest. CPU seems less of a restraint, but I conclude that 32G would improve the performance (for multitasking on desktop etc.)

OS: Ubuntu 24.04.1 x86_64
rustc 1.82.0

[nym21]

Very interesting thank you !
From what I see Memory and CPU usage seem pretty much constant but disk usage increases over time, I guess it makes sense.

Yeah, 16 GB of RAM would be feasible for a dedicated machine to this not a multitasking one.

Do you know how fast does it update ?
From: 2024-12-09 09:58:07 - Starting...
To: 2024-12-09 09:59:44 - Waiting for a new block...
This would give me a rough idea of your computer's power

Also there is an issue with databases growing and growing that's why I added a defragmentation option this fix that and does also make everything faster, you can run it with ./run.sh --first-defragment true
But please monitor it and stop the program once it's done and then run it normally as I didn't put the code at the right place and it will defragment every time the parser will find a new block.
If you prefer to wait, I'm working on the parser right now and will automate this to it easier

[xjmzx]

The old stats didn't have the timestamps saved but I can show the recent data.

2024-12-09 16:38:22 - ---
2024-12-09 16:38:22 - Configuration:
2024-12-09 16:38:22 - datadir: Some("/media/data/chain")
2024-12-09 16:38:22 - rpcconnect: None
2024-12-09 16:38:22 - rpcport: None
2024-12-09 16:38:22 - rpccookiefile: Some("/media/data/chain/.cookie")
2024-12-09 16:38:22 - rpcuser: None
2024-12-09 16:38:22 - rpcpassword: None
2024-12-09 16:38:22 - delay: None
2024-12-09 16:38:22 - dry_run: None
2024-12-09 16:38:22 - record_ram_usage: None
2024-12-09 16:38:22 - recompute_computed: None
2024-12-09 16:38:22 - first_defragment: None
2024-12-09 16:38:22 - ---
2024-12-09 16:38:22 - 873974 blocks found.
2024-12-09 16:38:22 - Starting...
2024-12-09 16:38:23 - Imported datasets
2024-12-09 16:38:23 - Imported databases
2024-12-09 16:38:23 - Imported states
2024-12-09 16:38:23 - min_datasets_inserted_last_height: Some(Height(873921))
2024-12-09 16:38:23 - min_datasets_inserted_last_date: Some(Date(2024-12-09))
2024-12-09 16:38:23 - Starting parsing at height: 873712
2024-12-09 16:38:25 - Processing 2024-12-08 (height: 873712)...
2024-12-09 16:39:16 - Iter through address_index_to_address_data: 51.879833 seconds
2024-12-09 16:39:17 - fetch kraken daily
2024-12-09 16:54:36 - Parsing group took 973.04 seconds (last height: 873857)
2024-12-09 16:54:36 - Exporting...
2024-12-09 16:54:39 - Datasets saved: 2.8755832 seconds
2024-12-09 16:54:40 - States saved: 0.26353914 seconds
2024-12-09 17:02:28 - Databases saved: 469.01193 seconds
2024-12-09 17:02:28 - Total save time: 471.89413 seconds

2024-12-09 17:02:28 - Processing 2024-12-09 (height: 873858)...
2024-12-09 17:08:35 - kraken: fetch 1mn
2024-12-09 17:12:34 - Parsing group took 605.7188 seconds (last height: 873974)
2024-12-09 17:12:34 - Computing datasets...
2024-12-09 17:12:42 - Computing datasets: 7.4711385 seconds
2024-12-09 17:12:42 - Exporting...
2024-12-09 17:12:49 - Datasets saved: 7.167408 seconds
2024-12-09 17:12:49 - Total save time: 7.167461 seconds

2024-12-09 17:12:56 - Waiting for a new block...
2024-12-09 17:12:56 - 873975 blocks found.
2024-12-09 17:12:56 - Starting...
2024-12-09 17:12:57 - Imported datasets
2024-12-09 17:12:57 - Imported databases
2024-12-09 17:12:57 - Imported states
2024-12-09 17:12:57 - min_datasets_inserted_last_height: Some(Height(873975))
2024-12-09 17:12:57 - min_datasets_inserted_last_date: Some(Date(2024-12-09))
2024-12-09 17:12:57 - Starting parsing at height: 873858
2024-12-09 17:12:58 - Processing 2024-12-09 (height: 873858)...
2024-12-09 17:13:55 - Iter through address_index_to_address_data: 56.839767 seconds
2024-12-09 17:13:55 - fetch kraken daily

Thanks for the note about defrag option, I will look into that. I think I'm going to upgrade to 32/32G RAM, when that is installed, I'll see how that goes.

[nym21]

So it takes you 34mn34 to update the datasets, I didn't expect that, that explains why it took you so long to parse the whole chain.

The same operation took 2mn40 on a base Mac mini M4 (so 16GB RAM) and a 2 TB external ssd (connected via thunderbolt 4):

2024-12-09 14:43:32 - Starting...
2024-12-09 14:43:33 - Imported datasets
2024-12-09 14:43:33 - Imported databases
2024-12-09 14:43:33 - Imported states
2024-12-09 14:43:33 - min_datasets_inserted_last_height: Some(Height(873959))
2024-12-09 14:43:33 - min_datasets_inserted_last_date: Some(Date(2024-12-09))
2024-12-09 14:43:33 - Starting parsing at height: 873712
2024-12-09 14:43:33 - Processing 2024-12-08 (height: 873712)...
2024-12-09 14:43:55 - Iter through address_index_to_address_data: 22.271402 seconds
2024-12-09 14:43:56 - fetch kraken daily
2024-12-09 14:44:47 - Parsing group took 74.189865 seconds (last height: 873857)
2024-12-09 14:44:47 - Exporting...
2024-12-09 14:44:47 - Datasets saved: 0.21582963 seconds
2024-12-09 14:44:47 - States saved: 0.08491621 seconds
2024-12-09 14:45:31 - Databases saved: 44.0669 seconds
2024-12-09 14:45:31 - Total save time: 44.284554 seconds

2024-12-09 14:45:31 - Processing 2024-12-09 (height: 873858)...
2024-12-09 14:46:06 - kraken: fetch 1mn
2024-12-09 14:46:06 - Parsing group took 34.95151 seconds (last height: 873959)
2024-12-09 14:46:06 - Computing datasets...
2024-12-09 14:46:09 - Computing datasets: 2.6337457 seconds
2024-12-09 14:46:09 - Exporting...
2024-12-09 14:46:11 - Datasets saved: 1.5759261 seconds
2024-12-09 14:46:11 - Total save time: 1.5759823 seconds

2024-12-09 14:46:11 - Waiting for a new block...

Your CPU seems to be pretty good so I think the issue is with IO, where do you store your data ?

[xjmzx]

That might be a bottleneck. I have the chain data on an internal Samsung SSD (Serial ATA-300,) that'll be slower than PCIe, right?

I should also mention that the kibo data is being written to a different drive (NVMe) which is the OS/boot disk. So, basically I guess the operations need to traverse the two.

[nym21]

ATA 300 is 300 MB/s, it's not fast but still, for the chain it's not the worst thing in the world as only very few (1 to like 3 I would say) blk files would be read to update

To give you an idea, the external drive that I use has 3-3.5GB/s reads and writes, tested via a program so actual not theoretical

[xjmzx]

That's quite helpful to work out the weaknesses of my system. I'm atleast sure CPU is not being utilized all that much.
Not just for using Kibo, but just thinking what system resources might be most important running different services.
Thanks for that.

[nym21]

No worries, happy to help !