use balanced ssds instead of spinning disks for remote-dev

[joaquincasares]

Description

Move to SSDs from HDDs for new remote-dev boxes for better random read/write patterns to contend with:

• the solana-validator writing about 10 Mb/s of data
• dockerd writing logs
• ganache writing blocks to disk
• postgres maintaining its database
• elasticsearch maintaining its database
• vscode-server's remote-ssh filewatchers
• git integration for certain p10k.sh shells
• normal terminal usage

Ideally, we would separate data for the solana-validator, ganache, postgres, and elasticsearch into a separate mounted/attached disk to lessen disk pressure on the boot disk allowing for a more responsive terminal and IDE experience.

If this change does not show vast improvement, we can try the pd-ssd disk type before considering a dedicated mounted disk as recommended by Docker:

• "Avoid storing application data in your container’s writable layer using storage drivers. This increases the size of your container and is less efficient from an I/O perspective than using volumes or bind mounts."
• "Instead, store data using volumes."

Background

After logging into my remote-dev machine, my terminal was laggy.

top showed about ~75% of the time the CPU was idle. free -m showed swap was off.

sar -b 1 was showing about 10 Mb/s being written out with about 1000 write transactions per second with 0 reads. However, pre-PR we were seeing sub-MB disk writes with an occasional burst to 100 M/s while post-PR we see a normal flow of at least 4 M/s with frequent bursts of 100 M/s.

sudo iotop -oP showed the majority of write requests was being spent on the solana-validator, dockerd, postgres, ganache, respectively.

vmstat 1 5 showed about 50k context switches per second.

iostat -x sda shows pre-PR r_await and w_await times (Vocabulary) of 4.37 and 9.74, respectively. Post-PR we see 0.70 and 5.44 on one node and 0.44 and 3.87 on the node with --fast, respectively. This shows that with the pd-balanced we tend to wait 90% less for reads and 60% less for writes.

pidstat -w 3 10   > /tmp/pidstat.out
pidstat -wt 3 10  > /tmp/pidstat-t.out
strace -c -f -p $PID

The above commands showed that node processes were seeing the majority of the cswch/s, however, strace showed they were stalling on futex calls implying node-based mutexes were the cuprit, which may be expected, but I'm unable to confirm without past data.

Some random glances samples that are representative of a live view are:

• pre-PR:
  • ctx_sw: 38k
  • inter: 25503
  • sw_int: 6154
  • iowait: Min:7.6 Mean:9.9 Max:12.4
• post-PR:
  • ctx_sw: 61k
  • inter: 42494
  • sw_int: 10934
  • iowait: Min:8.8 Mean:11.7 Max:16.3
• post-PR w/ --fast:
  • ctx_sw: 59k
  • inter: 43123
  • sw_int: 10556
  • iowait: Min:10.4 Mean:12.7 Max:18.7

-- Vocabulary

What glances highlights is that:

• We are able to context switch about 1.5x (61%) more often than when using pd-balanced.
• We were able to accept about 2x (72%) more hardware (network, disk) interrupts.
• We are able to accept about 2x (78%) more kernel-level interrupts.
• We are able to saturate the disks for longer with high iowait.
• Given that we've gone above 10 on our iowait stat, it's an indicator that we should have already had a plan in place to migrate when iowait was above 7.
• Sadly, even the cheaper pd-balanced SSDs are not enough to run our 4 databases + docker + VSCode server.

While our write throughputs would effectively double, we don't seem to be close to our current limitation of 400 MB/s.

However, while theoretical disk throughput could be similar between pd-standard and pd-balanced disks, the former are backed by HDDs while the latter are backed by SSDs at a cheaper price point than pd-ssd disks:

• https://cloud.google.com/compute/docs/disks#disk-types

Tests

Running A in split view for 4 machines with pd-balanced (pre-PR, post-PR, post-PR with --fast) and pd-ssd (post-PR with --fast).

How will this change be monitored? Are there sufficient logs?

#infra-troubleshooting

[cheran-senthil]

I think this is the associated linear card, https://linear.app/audius/issue/INF-41/prevent-remote-dev-becoming-100percent-full

I don't see how this PR reduces disk space usage. I have follow up PRs that drastically reduce Solana and Ganache disk usage already.

Moreover, I think it is possible that remote boxes are being setup without the default log size limit from https://github.com/AudiusProject/audius-docker-compose/blob/main/setup.sh#L21 that is usually only added when provisioned with --fast (?) if this is not added even so, then that is probably the cause of most of this trouble.

[dmanjunath]

i agree with @cheran-senthil that this probably doesn't solve the initial problem of us writing a ton of data. however i think this is a good change that should give us better read perf, so we should merge it. ultimately we gotta find out how to write less to disk.

[dmanjunath]

and yes, we should bake the docker log size change into the non fast create flow

[joaquincasares]

@cheran-senthil you're correct that the linear card doesn't quite match with this work, but both come from the same focus to focus on disk usage whether it be disk consumption or disk utilization. If need be, I can create a different ticket for disk utilization.

@dmanjunath what the stats in the description state is that we will get better performance, but even with pd-balanced SSDs, we're still hitting hit iowaits which is definitely going to cause problems as we start utilizing elasticsearch as well. Personally, I would hate to hurt morale by delivering a fix that will only work for a few weeks, before we start seeing the same level of pain we have been for the past couple of months.

I don't think the correct solution is pd-balanced due to these iowaits of 18, when they should be below 7-10. We're already getting data that cheapest SSD option works faster, but are under even higher disk contention.

For this to really benefit our team, we should either do 2 pd-balanced disks (what I and Docker recommend), or perhaps one pd-ssd (which will be costly, but I can test how well this does in this PR).

Thoughts?