Running Llama using VULKAN on an Arm Mali-G78AE GPU, the program hangs while waiting for a fence until it terminates after throwing an instance of 'vk::DeviceLostError'.

[warren-lei]

The project is built with aarch64-linux-gcc (GCC) 8.3.0 for aarch64 linux.

The command to run llama:
./llama-cli -m ggml-model-q4_k.gguf -c 512 -b 1024 -n 256 --keep 48 --repeat_penalty 1.0 --color -i -r "User:" -f chat-with-bob.txt -ngl 3

terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Device::waitForFences: ErrorDeviceLost
Aborted

This exception occurs in function ggml_vk_compute_forward at ctx->device->device.waitForFences:

    if (tensor == subctx.exit_tensor) {
        VK_CHECK(ctx->device->device.waitForFences({ ctx->fence }, true, UINT64_MAX), "ggml_vk_compute_forward waitForFences");
        //printf(">>>wait for vulkan fence done.\n");
        ctx->device->device.resetFences({ ctx->fence });

        // Do staging buffer copies
        for (auto& cpy : subctx.out_memcpys) {
            memcpy(cpy.dst, cpy.src, cpy.n);
        }
        subctx.in_memcpys.clear();
        subctx.out_memcpys.clear();
    }

[0cc4m]

That exception is the generic "something failed in the driver" error. In this case it probably means that the big block of pre-recorded GPU commands was too much for the hardware/software for whatever reason, so it crashed. This seems to be a common problem with phone GPUs, not sure how to work around it. Maybe someone experienced in mobile game development has some experience with this?

[warren-lei]

That exception is the generic "something failed in the driver" error. In this case it probably means that the big block of pre-recorded GPU commands was too much for the hardware/software for whatever reason, so it crashed. This seems to be a common problem with phone GPUs, not sure how to work around it. Maybe someone experienced in mobile game development has some experience with this?

If I run llama-cli with fp16 model ggml-model-f16.gguf, the device lost issue will not occur.
If I run llama-cli with Q4_k or Q8_0 model, it will throw an instance of 'vk::DeviceLostError'.