Dual spark support

[shaman3000]

Hi Folks,

How about dual dgx setup?

would be great to work with 2 x spark cluster

[hasso5703]

Thanks for asking. Not yet: everything here is built and measured on one box, and we only ship what we have run ourselves.

What two Sparks would change: linked by their ConnectX-7 port, two GB10 boxes can serve one model split across both (tensor or expert parallel in SGLang, one engine per box). That opens models that do not fit in one box's 128 GB, and NVIDIA documents the two-box setup for that reason. It also brings new failure modes (a link that drops, two boxes to update together, memory guards on both), so it would come as its own lane, with its own measurements, not as a flag on the current ones.

If you already run two Sparks linked this way, what do you serve on them, and would you test a branch when there is one? Numbers from a real pair would help us more than anything we could guess from one box.

[VincKay-max]

Hi @hasso5703,

Thanks for the update and for providing the read-only probe!
I'll be back in the office on Monday and plan to start with a fresh installation of Spark 1. I'll probably reinstall Spark 2 on Tuesday.

Afterwards, I'll run pair-probe.py on both systems and share the results here.

Thanks again for your work and support!

[hasso5703]

Hi @VincKay-max, thanks, Monday and Tuesday are perfect.

One change since our last message: on November 1 the project was renamed Alzeo, at github.com/hasso5703/alzeo. The one command for a fresh install is now:

curl -fsSL https://get.alzeo.ai | bash

It installs v1.41.0 into the same folder as before, ~/dgx-spark-qwen38, so the probe stays:

python3 ~/dgx-spark-qwen38/pair-probe.py

The old addresses keep working, as GitHub redirects them, and a box on an older version still updates from its cockpit.

We look forward to both outputs.

[VincKay-max]

Hi @hasso5703 ,

quick update on my dual DGX Spark setup.

I've now reinstalled Spark 1 and updated it to Alzeo v1.41.0 (6a8f2ec). The installation includes Qwen 27B and Flash, without OpenCode.

Spark 1 – current status:

• Hardware: Dell Pro Max GB10
• DGX OS: 7.6.0 / Ubuntu 24.04.5 LTS
• Kernel: 7.0.0-1024-nvidia
• NVIDIA driver: 580.178.04 / CUDA 13.0
• Docker: 29.6.2
• NVIDIA Container Toolkit: 1.20.1
• Alzeo: v1.41.0 (6a8f2ec)
• Active lane: 27B (SGLang)
• Flash lane: installed, currently inactive
• Time synchronization: OK

ConnectX-7 / RDMA:

The latest read-only pair probe now detects all four ConnectX-7 PCI functions.

• 2 ports UP at 200 Gbps
• 2 ports DOWN
• 2 RDMA devices ACTIVE at 200 Gbps
• mlx5_core and mlx5_ib loaded
• MTU 1500
• No IPv4 addresses assigned to the ConnectX-7 interfaces yet

[hasso5703]

Hi @VincKay-max, thanks, that is a clean start, and good to know the 27B lane runs on DGX OS 7.6.0 with kernel 7.0.0-1024.

Your ConnectX-7 lines look like one cable plugged in. Each physical port shows up as two interfaces, so two UP and two DOWN means one port linked at 200 Gbps. It only needs addresses, and only for the second stage (TP=2). The first stage does not use the cable.

When it suits you:

1. On Spark 2, the same one command, then the probe:

curl -fsSL https://get.alzeo.ai | bash
python3 ~/dgx-spark-qwen38/pair-probe.py

2. Please paste the full output of the probe from both boxes, not a summary. It is already masked, and it holds details the design needs, such as which interfaces are up and the lanes on each box.

3. Later, for the link itself: NVIDIA's "Connect multiple Sparks" playbook gives the link its addresses, with NVIDIA Sync's Cluster Assistant or by hand with netplan (https://build.nvidia.com/spark/connect-multiple-sparks). Then, on each box:

python3 ~/dgx-spark-qwen38/pair-probe.py --peer <the other box's address on that link>

On our side, the first-stage branch comes next: one endpoint for both boxes, each request routed by model name to the box that already serves that model, and both boxes in the cockpit. We will post here when it is ready to install and roll back.

[hasso5703]

Hi @VincKay-max, the first stage is in v1.42.0: two boxes behind one endpoint.

Each box keeps the lane it loads. A request that names the other box's model goes to that box, streamed or not, in the OpenAI and Anthropic dialects. GET /v1/models on either box lists both, and the cockpit's Now view shows the other box. The two boxes talk over TLS both ways on port 30002, each pinning the other's certificate, and neither holds the other's API key.

To try it once Spark 2 is set up:

1. Run the one-liner on both boxes, Spark 1 included, so that both run v1.42.0:

curl -fsSL https://get.alzeo.ai | bash

2. Make sure SSH works from one box to the other without a password, as step 4 of NVIDIA's connect-two-sparks playbook sets it up. ssh <the other box> true must work without a question.

3. On either box, with the other box's address as this box reaches it (your LAN or tailnet address is fine, the cable is not needed for this stage):

python3 ~/dgx-spark-qwen38/pair.py add <the other box's address>
python3 ~/dgx-spark-qwen38/pair.py status

add pairs both boxes at once and checks each way. When a check fails it leaves both boxes as they were and says what to fix.

It would help a lot if you could send:

• the output of pair.py add, and of pair.py status on both boxes
• the model each box serves, and one request through each box for the other box's model (a curl to /v1/chat/completions, or your usual client)
• anything that surprised you

The full guide is docs/pair.md. Splitting one model across both boxes over the ConnectX-7 link (TP=2) is the next stage, and pair-probe.py will help with that one.