Anyone here try with two clustered Sparks?

[jerkstorecaller]

It's gonna take me a couple of days to download the model, so I figured I'd ask about some things in the meantime.

I was going through the Quick Start Guide and read this: [Requirements] 400 GB of local NVMe — on one drive, or split across several: COLI_MODEL_DIRS=<dir2>[;<dir3>...] (in dev since #469) reads a distinct subset of the shards from each drive, so two 250 GB drives can host it together.

So the model can be read from two drives, great. Some performance numbers:

• The Spark's main drive reads at 1.4GB/s
• Two Sparks connected with the Connect-X7 cable transfer at 6.25 GB/s according to iperf3. So that's 4-5 times faster than the SSD.

So what happens when Spark 1 (the one running coli) mounts the Spark 2's drive as an NFS share and uses the network share as Directory #2? Do we get 2x the read speed?

I realize Spark 2 still has to read the shards off the SSD, but network transfer won't be a bottleneck. More importantly, Spark 2's Linux will keep an extra 128GB of shards in memory cache, so I feel like that could be huge speed boost right there, right?

Does that get us up from 3.33 t/s to a solid 5? Unfortunately I don't know enough about network shares to know if the NFS protocol has downsides that make this hope misguided.

I will be able to answer these questions myself eventually if no one else does. The curiosity is just eating me.

[nekromant]

I don't think it's a good idea, still more hops for the data and still only one spark doing actual work.

I'd more prefer to see some kind of routing logic, so that each spark holds a few hot shards in ram and each of them executes calculations. Just spark 1 does nothing, when a request gets a hit for a shard available in second's spark cache. It's not true 'tensor-parallel', but still might work for 2 concurrent requests.