v2.7.3 crashes when loading large models, where v2.5.1 did not

[dailysoftware]

Bug Report

GPT4ALL crashes without any warning when using a model with RAM requirements greater than 16 GB. But when I switch version to 2.5.1 or loading a model with RAM requirements under 8GB, there is no problem.

Steps to Reproduce

1.Run GPT4ALL
2.Choose model
3.GPT4All then crashes

Your Environment

• GPT4All version:2.7.3
• Operating System:Win11
• Chat model used (if applicable):nous-hermes-llama2-13b.Q4_0.gguf

[dailysoftware]

2.5.1 version can use various models, but the device can only use CPU and not GPU, with an error message indicating that GPU loading failed (out of VRAM). On the other hand, 2.7.3 version cannot use memory requirements of 16GB models, but it can use 8GB models and also use GPUs

[cebtenzzre]

How much RAM do you have? Do you think it is possible that GPT4All is running out of RAM (e.g. does it crash when you set the device to "CPU"), or is it really crashing when it runs out of VRAM? The latter is possible, but it would definitely be a bug and not an intentional occurrence.

[Syclusion]

I am having this issue as well, 4090 and 96gb of memory. Running on cpu fixes crash but runs slow af

[TREHAND-Christian]

I have the same problem, 80GB memory, NVIDIA RTX 3060.

QML debugging is enabled. Only use this in a safe environment.
[Debug] (Mon Apr 22 06:20:54 2024): deserializing chat "F:/AI/gpt4all/nomic.ai/GPT4All//gpt4all-3ca3afb4-8c17-4c97-8693-135477a84612.chat"
[Debug] (Mon Apr 22 06:20:54 2024): deserializing chats took: 4 ms
llama_new_context_with_model: max tensor size = 102.54 MB
llama.cpp: using Vulkan on NVIDIA GeForce RTX 3060
error loading model: Memory type index for buffer creation not found
llama_load_model_from_file_internal: failed to load model
LLAMA ERROR: failed to load model from F:/AI/gpt4all/nomic.ai/GPT4All/wizardcoder-python-34b-v1.0.Q4_0.gguf
GGML_ASSERT: C:\msys64\home\Jared\gpt4all-navarro\gpt4all-backend\llama.cpp-mainline\llama.cpp:552: data

[ItsCheif]

Issue seems to still exist on v2.8.0.

I've just got a large model that crashes GPT4ALL without warning, switched to CPU and it doesn't crash anymore. But it also just takes forever to write a single letter.

[securityopa]

I have the same problem with the latest version from flathub.

I have 128GB of RAM and I am using AMD Radeon 6800XT which is pretty fast in generating answers. But suddenly when the response is large, it crashes.