Can we reduce the startup time of the sapphire-dev container?

[CedarMist]

Currently with the mock backend it takes about a minute for sapphire-dev container to start on my laptop.

Ideally it should take less time, so it's worth doing some investigation to determine why it takes so long.

We probably shouldn't try things blindly, and instead try to instrument the container startup.

Some options are (unsure if they would make any difference):

• Reduce logging? (maybe disk I/O is causing delays?)
• Run only 1 compute node if possible.
• Removing additional sleeps
• Provision less accounts

There are ways of tracking startup time (e.g. a flame graph) which could be useful:

• https://github.com/mboussaa/docker-flame-graphs
• https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2023-1/profiling-in-docker-container.html
• https://stackoverflow.com/questions/53698972/how-to-do-profiling-inside-a-docker-container
• https://nanxiao.me/en/use-perf-and-flamegraph-to-profile-program-on-linux/
• https://gendignoux.com/blog/2019/11/09/profiling-rust-docker-perf.html

[ptrus]

Could you post some logs (with timestamps) of the startup part. I assume it would be somewhat easy to see what steps are using most of the time and how realistically it is reducing it down.

Reduce logging? (maybe disk I/O is causing delays?)

Most likely not the case.

Run only 1 compute node if possible.

Yeah should be possible and easy to setup via the fixture (I think the default uses 3 compute nodes).

Provision less accounts

How many accounts are currently provisioned? I assume all accounts are provisioned from the same "seed account" so it can only provision one new account per runtime block, so this could indeed slow the startup down. This could be optimized.

---

Some more ideas:

• if we find out there's no way to meaningfully reduce the startup-time further, the start-up phase could be moved into the build step of the docker container. We could, as part of the docker build phase, start and initialize the network state. On container startup it would then just need to resume the already initialized network - which should be fast (note that the oasis-node net-runner does already support this - resuming an already provisioned network).

• related to the above. Make the container non-ephemeral. With some updates the container could persist state across restarts, removing the need to re-provision/initialize the network on every restart.

[CedarMist]

I tend to run the containers with --rm otherwise I end up with a bazillion previously used Docker containers which makes it difficult to prune unused stuff in the docker cache.

I will focus on reducing initial startup time, reducing compute nodes to 1, provision less accounts (I'm doing 20, can reduce to 5). And see if anything else stands out as easy wins first before moving onto anything more in-depth.