New to LLMs. Are models small enough to work on much older hardware?

[blackdow1]

I have an old HP Compaq 6910p laptop (I think from 2013?) and I am looking to just toy around with it and see if it could run a small model. I ran this model: https://huggingface.co/unsloth/Qwen3-0.6B-GGUF on it with the Jan AI frontend. I did not tweak the settings and just ran it with the default settings. It struggles to run, but it gives a response after about 15 minutes. I am not familiar with LLMs, so was wondering if someone could help me in optimizing the llama.cpp settings to get the response time to under 5 minutes. And I am not sure if the choice of frontend is causing a slowdown. I had seen a post somewhere that LMStudio ran better than Jan AI.

[cookesan]

The frontend is probably not the main thing to optimize first. On a laptop from that era, most of the time will be spent in the local inference backend and memory subsystem, so I would first take Jan out of the test and measure a direct llama.cpp run.

Use the smallest quantized GGUF you are comfortable with, keep the prompt/context small, and cap the generated output while testing. For example:

llama-cli -m /path/to/Qwen3-0.6B-....gguf -t 1 -c 1024 -n 128 --perf -p "Write a short reply about..."

Then repeat with -t 2. The performance guide specifically calls out that too many generation threads can oversaturate the CPU; it recommends starting at 1, increasing until performance stops improving, then backing down. On a Core 2 Duo-era machine, the best value may be lower than expected.

The other useful checks are:

• Use a small quant such as a Q4_* file, or smaller if the quality is acceptable.
• Keep -n low while testing; generating 128 tokens is a very different target from generating a long answer.
• Keep -c modest. Large context sizes add memory pressure.
• If the machine is swapping, try closing other apps. --mlock can help only if there is enough RAM to keep the model resident.
• -ngl only helps if you built with a supported GPU backend and the hardware can actually offload layers.

If the direct llama-cli timing is still many minutes for a short prompt and capped output, changing frontends is unlikely to turn it into a sub-five-minute setup. At that point the realistic options are a smaller quant/model, shorter generations, or newer hardware.

[Atomic-Germ]

How much ram do you have and what backend are you trying to use? You're going to have a bad time with anything but the CPU back. I'd suggest compiling on that system if possible, too. If you have the stock 4gb, I think you're already getting what you'll be able to get. I doubt LM Studio is helping.

Here's a few models that might work out for you; compile with cmake -B build && cmake --build build -j and nothing else:

$ llama-bench -hf Tralalabs/OpenClaude-1.7B-Merged-Q3_K_M-GGUF
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen3 1.7B Q3_K - Medium       | 890.34 MiB |     1.72 B | CPU        |       6 |           pp512 |        149.23 ± 3.87 |
| qwen3 1.7B Q3_K - Medium       | 890.34 MiB |     1.72 B | CPU        |       6 |           tg128 |         38.55 ± 0.63 |

llama-bench -hf x1nx3r/Llama-3.2-3B-thinking-100K-Q3_K_M-GGUF
Downloading checkpoint-11500-Q3_K_M.gguf ─────────────────────────── 100%
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 3B Q3_K - Medium         |   1.56 GiB |     3.21 B | CPU        |       6 |           pp512 |         78.64 ± 0.17 |
| llama 3B Q3_K - Medium         |   1.56 GiB |     3.21 B | CPU        |       6 |           tg128 |         22.42 ± 0.16 |

llama-bench -hf x1nx3r/Llama-3.2-3B-thinking-100K-Q2_K-GGUF
Downloading checkpoint-11500-Q2_K.gguf ───────────────────────────── 100%
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 3B Q2_K - Medium         |   1.26 GiB |     3.21 B | CPU        |       6 |           pp512 |         86.09 ± 0.59 |
| llama 3B Q2_K - Medium         |   1.26 GiB |     3.21 B | CPU        |       6 |           tg128 |         26.81 ± 0.37 |

llama-bench -hf Anonyme843/Ternary_Bonsai_1.7b_Multi-GGUF:Q2_K
Downloading ternary-bonsai-1.7b-q2_k.gguf ────────────────────────── 100%
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen3 1.7B Q2_K - Medium       | 735.66 MiB |     1.72 B | CPU        |       6 |           pp512 |        168.99 ± 4.46 |
| qwen3 1.7B Q2_K - Medium       | 735.66 MiB |     1.72 B | CPU        |       6 |           tg128 |         45.43 ± 0.23 |

Those should all fit in the stock ram on your computer. If you can get 8gb or 16gb in there, that would help a lot for quality, but these are all good for speed and size. A small model with a higher quantized gguf, in my experience, is better than a large on with a lower quant, depending on the model I'd gather.

[blackdow1]

Thanks for the replies. I really appreciate it. I had recently installed Linux on the spare laptop, of which I am a beginner user of Linux as well. The memory on it is of 4GB while the swap is of 4 as well. I did the cmake for the cpu build. I used the 4-bit IQ4_XS variant of the model. I used the llama-cli -m /path/to/Qwen3-0.6B-....gguf -t 1 -c 1024 -n 128 --perf -p "Write a short reply about..." command and it gave a result of Prompt: 1.3 t/s and Generation: 1 t/s in about 4 minutes. Is it standard for the output to be "[Start thinking] Example of thoughts, ramblings and indecisiveness" included in the response? This is a good base for me to start with though. Thanks again for the replies and help.

[aryanrao7]

On that class of laptop the frontend is probably not the main bottleneck. The HP Compaq 6910p is much older than 2013-era hardware and is usually a Core 2 Duo class machine, so llama.cpp will be running with very limited SIMD and memory bandwidth. A 0.6B model can fit, but long generations will still be slow.

Things I would try first:

# Test the raw llama.cpp path without Jan/LM Studio overhead
./llama-cli \
  -m Qwen3-0.6B-GGUF-file.gguf \
  -t 2 \
  -c 512 \
  -b 16 \
  -ub 16 \
  -n 96 \
  -p "Answer briefly: what is llama.cpp?"

Then compare with:

./llama-bench -m Qwen3-0.6B-GGUF-file.gguf -t 1,2 -p 64 -n 64

The settings that matter most on that machine are:

• keep -t at the physical core count, likely 2; more threads can make it worse
• keep context small, e.g. -c 256 or -c 512, not thousands of tokens
• keep -n low if your target is under 5 minutes; generation length dominates wall time
• use the smallest quantization you are comfortable with, e.g. Q2/Q3 instead of Q4/Q5
• run llama-cli directly once to measure baseline before judging any frontend

If the model currently takes 15 minutes with default frontend settings, getting under 5 minutes may require all three: smaller quantization, shorter context, and a hard output cap. If llama-bench still shows very low tokens/sec, the remaining limit is just the CPU, not Jan vs LM Studio.