Raspberry Pi 5 cannot serve models with `llama.cpp`

[znmeb]

Issue Description

On a Raspberry Pi 5, ramalama appears to be using a llama.cpp executable compiled with Vulkan enabled. That doesn't work; see this issue for some details.

ggml-org/llama.cpp#9801

Steps to reproduce the issue

1. Get a Raspberry Pi 5 - this appears to be hardware dependent.
2. Install ramalama. On my system, I have Homebrew, and your curl installer used it to install ramalama from Homebrew.
3. Run ramalama --runtime llama.cpp serve mistral:7b`

Describe the results you received

❯ ramalama --runtime llama.cpp serve mistral:7b
load_backend: loaded RPC backend from /usr/bin/libggml-rpc.so
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = V3D 7.1.10.2 (V3DV Mesa) | uma: 1 | fp16: 0 | bf16: 0 | warp size: 16 | shared memory: 16384 | int dot: 0 | matrix cores: none
load_backend: loaded Vulkan backend from /usr/bin/libggml-vulkan.so
load_backend: loaded CPU backend from /usr/bin/libggml-cpu-armv8.2_2.so
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 1 (a69d54f) with GNU 15.2.1 for Linux aarch64
system info: n_threads = 4, n_threads_batch = 4, total_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

Running without SSL
init: using 6 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/mnt/models/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
ggml_vulkan: Error: Shared memory size too small for matrix multiplication.
llama_model_load: error loading model: Shared memory size too small for matrix multiplication.
llama_model_load_from_file_impl: failed to load model
llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
llama_params_fit: fitting params to free memory took 0.07 seconds
llama_model_load_from_file_impl: using device Vulkan0 (V3D 7.1.10.2) (unknown id) - 4096 MiB free
llama_model_loader: loaded meta data with 29 key-value pairs and 291 tensors from /mnt/models/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = Mistral-7B-Instruct-v0.3
llama_model_loader: - kv   2:                          llama.block_count u32              = 32
llama_model_loader: - kv   3:                       llama.context_length u32              = 32768
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   8:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                          general.file_type u32              = 15
llama_model_loader: - kv  11:                           llama.vocab_size u32              = 32768
llama_model_loader: - kv  12:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32768]   = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32768]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32768]   = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  22:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  23:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv  24:               general.quantization_version u32              = 2
llama_model_loader: - kv  25:                      quantize.imatrix.file str              = /models/Mistral-7B-Instruct-v0.3-GGUF...
llama_model_loader: - kv  26:                   quantize.imatrix.dataset str              = /training_data/calibration_data.txt
llama_model_loader: - kv  27:             quantize.imatrix.entries_count i32              = 224
llama_model_loader: - kv  28:              quantize.imatrix.chunks_count i32              = 228
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 4.07 GiB (4.83 BPW) 
load: 0 unused tokens
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 771
load: token to piece cache size = 0.1731 MB
print_info: arch                  = llama
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 32768
print_info: n_embd                = 4096
print_info: n_embd_inp            = 4096
print_info: n_layer               = 32
print_info: n_head                = 32
print_info: n_head_kv             = 8
print_info: n_rot                 = 128
print_info: n_swa                 = 0
print_info: is_swa_any            = 0
print_info: n_embd_head_k         = 128
print_info: n_embd_head_v         = 128
print_info: n_gqa                 = 4
print_info: n_embd_k_gqa          = 1024
print_info: n_embd_v_gqa          = 1024
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-05
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 0.0e+00
print_info: n_ff                  = 14336
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = 0
print_info: rope type             = 0
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: n_ctx_orig_yarn       = 32768
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 7B
print_info: model params          = 7.25 B
print_info: general.name          = Mistral-7B-Instruct-v0.3
print_info: vocab type            = SPM
print_info: n_vocab               = 32768
print_info: n_merges              = 0
print_info: BOS token             = 1 '<s>'
print_info: EOS token             = 2 '</s>'
print_info: UNK token             = 0 '<unk>'
print_info: LF token              = 781 '<0x0A>'
print_info: EOG token             = 2 '</s>'
print_info: max token length      = 48
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)

Describe the results you expected

I expected it to serve the model.

ramalama info output

❯ ramalama info
{
    "Accelerator": "none",
    "Config": {
        "benchmarks": {},
        "provider": {
            "openai": {}
        },
        "settings": {
            "config_files": [
                "/home/linuxbrew/.linuxbrew/Cellar/ramalama/0.18.0/libexec/share/ramalama/ramalama.conf"
            ]
        },
        "user": {}
    },
    "Engine": {
        "Info": {
            "host": {
                "arch": "arm64",
                "buildahVersion": "1.43.0",
                "cgroupControllers": [
                    "cpu",
                    "pids"
                ],
                "cgroupManager": "systemd",
                "cgroupVersion": "v2",
                "conmon": {
                    "package": "Unknown",
                    "path": "/home/linuxbrew/.linuxbrew/bin/conmon",
                    "version": "conmon version 2.2.1, commit: "
                },
                "cpuUtilization": {
                    "idlePercent": 99.07,
                    "systemPercent": 0.21,
                    "userPercent": 0.71
                },
                "cpus": 4,
                "databaseBackend": "sqlite",
                "distribution": {
                    "codename": "trixie",
                    "distribution": "debian",
                    "version": "13"
                },
                "eventLogger": "journald",
                "freeLocks": 2043,
                "hostname": "partch",
                "idMappings": {
                    "gidmap": [
                        {
                            "container_id": 0,
                            "host_id": 1000,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 100000,
                            "size": 65536
                        }
                    ],
                    "uidmap": [
                        {
                            "container_id": 0,
                            "host_id": 1000,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 100000,
                            "size": 65536
                        }
                    ]
                },
                "kernel": "6.12.75+rpt-rpi-2712",
                "linkmode": "dynamic",
                "logDriver": "journald",
                "memFree": 7946944512,
                "memTotal": 17006772224,
                "networkBackend": "netavark",
                "networkBackendInfo": {
                    "backend": "netavark",
                    "defaultNetwork": "podman",
                    "dns": {
                        "package": "Unknown",
                        "path": "/home/linuxbrew/.linuxbrew/opt/podman/libexec/podman/aardvark-dns",
                        "version": "aardvark-dns 1.17.0"
                    },
                    "package": "Unknown",
                    "path": "/home/linuxbrew/.linuxbrew/opt/podman/libexec/podman/netavark",
                    "version": "netavark 1.17.2"
                },
                "ociRuntime": {
                    "name": "crun",
                    "package": "Unknown",
                    "path": "/home/linuxbrew/.linuxbrew/bin/crun",
                    "version": "crun version 1.27\ncommit: a718a92cc9a94955a5a550b6fdec1378c247ec50\nrundir: /run/user/1000/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL"
                },
                "os": "linux",
                "pasta": {
                    "executable": "/home/linuxbrew/.linuxbrew/bin/pasta",
                    "package": "Unknown",
                    "version": "pasta 2026_01_20.386b5f5\nCopyright Red Hat\nGNU General Public License, version 2 or later\n  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
                },
                "remoteSocket": {
                    "exists": true,
                    "path": "/run/user/1000/podman/podman.sock"
                },
                "rootlessNetworkCmd": "pasta",
                "security": {
                    "apparmorEnabled": false,
                    "capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
                    "rootless": true,
                    "seccompEnabled": true,
                    "seccompProfilePath": "",
                    "selinuxEnabled": false
                },
                "serviceIsRemote": false,
                "slirp4netns": {
                    "executable": "/home/linuxbrew/.linuxbrew/bin/slirp4netns",
                    "package": "Unknown",
                    "version": "slirp4netns version 1.3.3\ncommit: 944fa94090e1fd1312232cbc0e6b43585553d824\nlibslirp: 4.9.1\nSLIRP_CONFIG_VERSION_MAX: 6\nlibseccomp: 2.6.0"
                },
                "swapFree": 2147467264,
                "swapTotal": 2147467264,
                "uptime": "1h 11m 47.00s (Approximately 0.04 days)",
                "variant": "v8"
            },
            "plugins": {
                "authorization": null,
                "log": [
                    "k8s-file",
                    "none",
                    "passthrough",
                    "journald"
                ],
                "network": [
                    "bridge",
                    "macvlan",
                    "ipvlan"
                ],
                "volume": [
                    "local"
                ]
            },
            "registries": {
                "search": [
                    "docker.io"
                ]
            },
            "store": {
                "configFile": "/home/znmeb/.config/containers/storage.conf",
                "containerStore": {
                    "number": 1,
                    "paused": 0,
                    "running": 0,
                    "stopped": 1
                },
                "graphDriverName": "overlay",
                "graphOptions": {},
                "graphRoot": "/home/znmeb/.local/share/containers/storage",
                "graphRootAllocated": 252026511360,
                "graphRootUsed": 116655665152,
                "graphStatus": {
                    "Backing Filesystem": "extfs",
                    "Native Overlay Diff": "true",
                    "Supports d_type": "true",
                    "Supports shifting": "false",
                    "Supports volatile": "true",
                    "Using metacopy": "false"
                },
                "imageCopyTmpDir": "/var/tmp",
                "imageStore": {
                    "number": 4
                },
                "runRoot": "/run/user/1000/containers",
                "transientStore": false,
                "volumePath": "/home/znmeb/.local/share/containers/storage/volumes"
            },
            "version": {
                "APIVersion": "5.8.1",
                "BuildOrigin": "brew",
                "Built": 1773235864,
                "BuiltTime": "Wed Mar 11 06:31:04 2026",
                "GitCommit": "",
                "GoVersion": "go1.26.1",
                "Os": "linux",
                "OsArch": "linux/arm64",
                "Version": "5.8.1"
            }
        },
        "Name": "podman"
    },
    "Image": "quay.io/ramalama/ramalama:0.18",
    "RagImage": "quay.io/ramalama/ramalama-rag:0.18",
    "Runtimes": {
        "Available": [
            "llama.cpp",
            "mlx",
            "vllm"
        ],
        "Default": "llama.cpp"
    },
    "Selinux": false,
    "Shortnames": {
        "Files": [
            "/home/linuxbrew/.linuxbrew/Cellar/ramalama/0.18.0/libexec/share/ramalama/shortnames.conf"
        ],
        "Names": {
            "cerebrum": "huggingface://froggeric/Cerebrum-1.0-7b-GGUF/Cerebrum-1.0-7b-Q4_KS.gguf",
            "deepseek": "ollama://deepseek-r1",
            "dragon": "huggingface://llmware/dragon-mistral-7b-v0/dragon-mistral-7b-q4_k_m.gguf",
            "gemma3": "hf://ggml-org/gemma-3-4b-it-GGUF",
            "gemma3:12b": "hf://ggml-org/gemma-3-12b-it-GGUF",
            "gemma3:1b": "hf://ggml-org/gemma-3-1b-it-GGUF/gemma-3-1b-it-Q4_K_M.gguf",
            "gemma3:27b": "hf://ggml-org/gemma-3-27b-it-GGUF",
            "gemma3:4b": "hf://ggml-org/gemma-3-4b-it-GGUF",
            "gemma3n": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-Q8_0.gguf",
            "gemma3n:e2b": "hf://ggml-org/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-Q8_0.gguf",
            "gemma3n:e2b-it-f16": "hf://ggml-org/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-f16.gguf",
            "gemma3n:e2b-it-q8_0": "hf://ggml-org/gemma-3n-E2B-it-GGUF/gemma-3n-E2B-it-Q8_0.gguf",
            "gemma3n:e4b": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-Q8_0.gguf",
            "gemma3n:e4b-it-f16": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-f16.gguf",
            "gemma3n:e4b-it-q8_0": "hf://ggml-org/gemma-3n-E4B-it-GGUF/gemma-3n-E4B-it-Q8_0.gguf",
            "gpt-5.1": "openai://gpt-5.1-2025-11-13",
            "gpt-oss": "hf://ggml-org/gpt-oss-20b-GGUF",
            "gpt-oss:120b": "hf://ggml-org/gpt-oss-120b-GGUF",
            "gpt-oss:20b": "hf://ggml-org/gpt-oss-20b-GGUF",
            "granite": "ollama://granite3.1-dense",
            "granite-be-3.0:1b": "hf://taronaeo/Granite-3.0-1B-A400M-Instruct-BE-GGUF/granite-3.0-1b-a400m-instruct-be.Q2_K.gguf",
            "granite-be-3.3:2b": "hf://taronaeo/Granite-3.3-2B-Instruct-BE-GGUF/granite-3.3-2b-instruct-be.Q4_K_M.gguf",
            "granite-lab-7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "granite-lab-8b": "huggingface://ibm-granite/granite-3.3-8b-instruct-GGUF/granite-3.3-8b-instruct-Q4_K_M.gguf",
            "granite-lab:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "granite:2b": "ollama://granite3.1-dense:2b",
            "granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "granite:8b": "ollama://granite3.1-dense:8b",
            "hermes": "huggingface://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF/Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf",
            "ibm/granite": "ollama://granite3.1-dense:8b",
            "ibm/granite:2b": "ollama://granite3.1-dense:2b",
            "ibm/granite:7b": "huggingface://instructlab/granite-7b-lab-GGUF/granite-7b-lab-Q4_K_M.gguf",
            "ibm/granite:8b": "ollama://granite3.1-dense:8b",
            "merlinite": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "merlinite-lab-7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "merlinite-lab:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "merlinite:7b": "huggingface://instructlab/merlinite-7b-lab-GGUF/merlinite-7b-lab-Q4_K_M.gguf",
            "mistral": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral-small3.1": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ2_M.gguf",
            "mistral-small3.1:24b": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF/mistralai_Mistral-Small-3.1-24B-Instruct-2503-IQ2_M.gguf",
            "mistral:7b": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral:7b-v1": "huggingface://TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q5_K_M.gguf",
            "mistral:7b-v2": "huggingface://TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q4_K_M.gguf",
            "mistral:7b-v3": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf",
            "mistral_code_16k": "huggingface://TheBloke/Mistral-7B-Code-16K-qlora-GGUF/mistral-7b-code-16k-qlora.Q4_K_M.gguf",
            "mistral_codealpaca": "huggingface://TheBloke/Mistral-7B-codealpaca-lora-GGUF/mistral-7b-codealpaca-lora.Q4_K_M.gguf",
            "mixtao": "huggingface://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF/MixTAO-7Bx2-MoE-Instruct-v7.0.Q4_K_M.gguf",
            "openchat": "huggingface://TheBloke/openchat-3.5-0106-GGUF/openchat-3.5-0106.Q4_K_M.gguf",
            "openorca": "huggingface://TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf",
            "phi2": "huggingface://MaziyarPanahi/phi-2-GGUF/phi-2.Q4_K_M.gguf",
            "qwen2.5vl": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:2b": "hf://ggml-org/Qwen2.5-VL-2B-Instruct-GGUF",
            "qwen2.5vl:32b": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:3b": "hf://ggml-org/Qwen2.5-VL-3B-Instruct-GGUF",
            "qwen2.5vl:7b": "hf://ggml-org/Qwen2.5-VL-7B-Instruct-GGUF",
            "smollm:135m": "hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF",
            "smolvlm": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "smolvlm:256m": "hf://ggml-org/SmolVLM-256M-Instruct-GGUF",
            "smolvlm:2b": "hf://ggml-org/SmolVLM-Instruct-GGUF",
            "smolvlm:500m": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "stories-be:260k": "hf://taronaeo/tinyllamas-BE/stories260K-be.gguf",
            "tiny": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
            "tinyllama": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
        },
        "Sources": [
            "/home/linuxbrew/.linuxbrew/Cellar/ramalama/0.18.0/libexec/share/ramalama/shortnames.conf"
        ]
    },
    "Store": "/home/znmeb/.local/share/ramalama",
    "UseContainer": true,
    "Version": "0.18.0"
}

Upstream Latest Release

Yes

Additional environment details

Raspberry Pi 5, 16 GB RAM, PiOS 64 bit, Homebrew.

Additional information

This is a known problem with the Raspberry Pi - you have to use a llama.cpp executable compiled without Vulkan on them.

[olliewalsh]

Have you tried with --runtime-args='-fit off' as the llama-server log mentions?

[olliewalsh]

Or does -ngl 0 avoid the issue?

[olliewalsh]

If neither works then can use the upstream llama.cpp container image with ramalama serve --image ghcr.io/ggml-org/llama.cpp:full mistral:7b. The full image is CPU only AFAIK.

[znmeb]

Where is this documented? I got ramalama from Homebrew and only found the root cause of the issue via a Google search leading to a closed bug against llama.cpp on GitHub! I get that AI technology is moving rapidly but the end result is that people won't use ramalama on a Pi 5. In other words, why are big names like qwen3.5 and gemma4 hyping models capable of useful deployment in 8 GB or even 4 GB single-board computers if they won't actually run there?

[olliewalsh]

Where is this documented?

The --image option? https://ramalama.ai/docs/commands/ramalama/serve/#--imageimage

why are big names like qwen3.5 and gemma4 hyping models capable of useful deployment in 8 GB or even 4 GB single-board computers if they won't actually run there?

Do you have a reference? I've not seen qwen (Alibaba) and gemma4 (Google) claim this. The models probably can run on a Pi5, but would be extremely slow.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.