[v0.17.1] Startlux 4B 9B Instant Q40 on 4 x Raspberry Pi 5 8GB

[b4rtaz]

Setup:

[🔀 Sagemcom Funbox6 Router]
      | | | |
      | | | |_______ 🔸 raspi0 (ROOT)     10.0.0.1
      | | |_________ 🔹 raspi2 (WORKER 1) 10.0.0.2
      | |___________ 🔹 raspi3 (WORKER 2) 10.0.0.3
      |_____________ 🔹 raspi4 (WORKER 3) 10.0.0.10

Device: 4 x Raspberry Pi 5 8GB
Distributed Llama version: 0.17.1

dllama_model_startlux_4b_instant_q40

Setup	Tok/s	Response time (seconds)
1 x Raspberry PI 5	4.72	20.993
2 x Raspberry PI 5	9.57	10.346
4 x Raspberry PI 5	17.50	5.656

dllama_model_startlux_9b_instant_q40

b4rtaz@raspi0:~/Dev/distributed-llama $ ./systemone-api \
--decision-config ~/Dev/system1/decision_config.json \
--model ~/Dev/system1/dllama_model_startlux_9b_instant_q40.m \
--tokenizer ~/Dev/system1/dllama_tokenizer_startlux_9b_instant.t \
--nbatches 256 \
--max-seq-len 256 \
--buffer-float-type q80 \
--nthreads 4 \
--workers 10.0.0.2:9999 10.0.0.3:9999 10.0.0.10:9999 --host 0.0.0.0

Setup	Tok/s	Response time (seconds)
1 x Raspberry PI 5	-	-
2 x Raspberry PI 5	4.66	21.260
4 x Raspberry PI 5	9.35	10.637

---

All above tests are performed on this request to systemone api:

    const body = {
        state: {
            customer_message: 'I was charged twice for my subscription.'
        },
        questions: {
            department: {
                type: 'choice',
                instructions: 'Choose the correct support department.',
                criteria: {
                    billing: 'Payments, invoices, refunds, or duplicate charges',
                    shipping: 'Delivery status or damaged shipments',
                    technical: 'Software bugs or account access problems'
                }
            }
        },
        none_check: false,
        return_details: true
    };

Example response:

b4rtaz@b4rtazs-MacBook-Pro examples % HOST=raspi0.local node systemone-client.js
{
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.9982326030731201,
      "probabilities": {
        "billing": 0.9988217353820801,
        "shipping": 0.00020235756528563797,
        "technical": 0.000975881761405617
      },
      "details": {
        "first_pass": {
          "probabilities": {
            "billing": 0.9988217353820801,
            "shipping": 0.00020235756528563797,
            "technical": 0.000975881761405617
          },
          "top": 0.9988217353820801,
          "ms": 21260.084381
        },
        "thought": {
          "used": false,
          "tokens": 0,
          "ended": "skipped",
          "ms": 0
        },
        "after_thought": null,
        "gate": null,
        "mix": null,
        "total_ms": 21298.972681
      }
    }
  },
  "usage": {
    "input_tokens": 99,
    "output_tokens": 0
  },
  "model": "dllama_model_startlux_9b_instant_q40.m",
  "timing_ms": {
    "wall": 21299.161626,
    "first_pass_sum": 21260.084381,
    "thought_span": 0,
    "after_thought_sum": 0
  }
}

Download Q40 model here.