Bug: unsupported op 'MUL_MAT' on bf16 but not f16 on SmolLM #499
Labels
Comments
|
There's no Apple Metal support for BF16 yet. Try passing the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Contact Details
jason.t.stillerman@gmail.com
What happened?
I am trying to convert the SmolLM models into llamafiles in bf16 and there is something weird happening. I thought I had this working earlier today but maybe that was in f16. Looks like @jart has this working in #495 but maybe that wasn't for metal?
It hangs here and the terminal must be force quitted. Ctrl-C does nothing
Here is my setup
Full logs in the "relevant log output" section
Version
llamafile v0.8.9
M2 MacBook Air 2023 Running Sonoma 14.0
What operating system are you seeing the problem on?
Mac
Relevant log output
(justine) jts@Jasons-MacBook-Air justine % llamafile -m smol-135-bf.gguf Apple Metal GPU support successfully loaded {"build":1500,"commit":"a30b324","function":"server_cli","level":"INFO","line":2869,"msg":"build info","tid":"34364712224","timestamp":1721684277} {"function":"server_cli","level":"INFO","line":2872,"msg":"system info","n_threads":4,"n_threads_batch":-1,"system_info":"AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | ","tid":"34364712224","timestamp":1721684277,"total_threads":8} llama_model_loader: loaded meta data with 30 key-value pairs and 272 tensors from smol-135-bf.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = SmolLM 135M llama_model_loader: - kv 3: general.basename str = SmolLM llama_model_loader: - kv 4: general.size_label str = 135M llama_model_loader: - kv 5: general.license str = apache-2.0 llama_model_loader: - kv 6: general.languages arr[str,1] = ["en"] llama_model_loader: - kv 7: general.datasets arr[str,1] = ["HuggingFaceTB/smollm-corpus"] llama_model_loader: - kv 8: llama.block_count u32 = 30 llama_model_loader: - kv 9: llama.context_length u32 = 2048 llama_model_loader: - kv 10: llama.embedding_length u32 = 576 llama_model_loader: - kv 11: llama.feed_forward_length u32 = 1536 llama_model_loader: - kv 12: llama.attention.head_count u32 = 9 llama_model_loader: - kv 13: llama.attention.head_count_kv u32 = 3 llama_model_loader: - kv 14: llama.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 15: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 16: general.file_type u32 = 32 llama_model_loader: - kv 17: llama.vocab_size u32 = 49152 llama_model_loader: - kv 18: llama.rope.dimension_count u32 = 64 llama_model_loader: - kv 19: tokenizer.ggml.add_space_prefix bool = false llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 22: tokenizer.ggml.pre str = smollm llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "<|im_start|>", "<|... llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,49152] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ... llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,48900] = ["Ġ t", "Ġ a", "i n", "h e", "Ġ Ġ... llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 0 llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 0 llama_model_loader: - kv 28: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 29: general.quantization_version u32 = 2 llama_model_loader: - type f32: 61 tensors llama_model_loader: - type bf16: 211 tensors llm_load_vocab: special tokens definition check successful ( 17/49152 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = BPE llm_load_print_meta: n_vocab = 49152 llm_load_print_meta: n_merges = 48900 llm_load_print_meta: n_ctx_train = 2048 llm_load_print_meta: n_embd = 576 llm_load_print_meta: n_head = 9 llm_load_print_meta: n_head_kv = 3 llm_load_print_meta: n_layer = 30 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_embd_head_k = 64 llm_load_print_meta: n_embd_head_v = 64 llm_load_print_meta: n_gqa = 3 llm_load_print_meta: n_embd_k_gqa = 192 llm_load_print_meta: n_embd_v_gqa = 192 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 1536 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 0 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 2048 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = BF16 llm_load_print_meta: model params = 134.52 M llm_load_print_meta: model size = 256.63 MiB (16.00 BPW) llm_load_print_meta: general.name = SmolLM 135M llm_load_print_meta: BOS token = 0 '<|endoftext|>' llm_load_print_meta: EOS token = 0 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '<|endoftext|>' llm_load_print_meta: LF token = 143 'Ä' llm_load_print_meta: EOT token = 2 '<|im_end|>' llm_load_tensors: ggml ctx size = 0.32 MiB ggml_backend_metal_log_allocated_size: allocated buffer, size = 202.66 MiB, ( 202.72 / 10922.67) llm_load_tensors: offloading 30 repeating layers to GPU llm_load_tensors: offloaded 30/31 layers to GPU llm_load_tensors: Metal buffer size = 202.65 MiB llm_load_tensors: CPU buffer size = 256.63 MiB ................................................................... llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 ggml_metal_init: allocating ggml_metal_init: found device: Apple M2 ggml_metal_init: picking default device: Apple M2 ggml_metal_init: default.metallib not found, loading from source ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil ggml_metal_init: loading '/Users/jts/.llamafile/v/0.8.9/ggml-metal.metal' ggml_metal_init: GPU name: Apple M2 ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008) ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003) ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001) ggml_metal_init: simdgroup reduction support = true ggml_metal_init: simdgroup matrix mul. support = true ggml_metal_init: hasUnifiedMemory = true ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB llama_kv_cache_init: Metal KV buffer size = 11.25 MiB llama_new_context_with_model: KV self size = 11.25 MiB, K (f16): 5.62 MiB, V (f16): 5.62 MiB llama_new_context_with_model: CPU output buffer size = 0.19 MiB llama_new_context_with_model: Metal compute buffer size = 14.50 MiB llama_new_context_with_model: CPU compute buffer size = 97.13 MiB llama_new_context_with_model: graph nodes = 966 llama_new_context_with_model: graph splits = 3 ggml_metal_graph_compute_block_invoke: error: unsupported op 'MUL_MAT' GGML_ASSERT: /Users/jts/.llamafile/v/0.8.9/ggml-metal.m:946: !"unsupported op" ggml_metal_graph_compute_block_invoke: error: unsupported op 'MUL_MAT' GGML_ASSERT: /Users/jts/.llamafile/v/0.8.9/ggml-metal.m:946: !"unsupported op" ggml_metal_graph_compute_block_invoke: error: unsupported op 'MUL_MAT' GGML_ASSERT: /Users/jts/.llamafile/v/0.8.9/ggml-metal.m:946: !"unsupported op" ggml_metal_graph_compute_block_invoke: error: unsupported op 'MUL_MAT' GGML_ASSERT: /Users/jts/.llamafile/v/0.8.9/ggml-metal.m:946: !"unsupported op"The text was updated successfully, but these errors were encountered: