Is llama-server secure to use from multiple users and rag from multiple users as service? Do it save the memory and data?

[biapar]

Is llama-server secure to use from multiple users and rag from multiple users as service? Do it save the memory and data?

[pballada]

Short answers, from the server README (tools/server/README.md).

Does it save memory or data?

Not by default. llama-server is essentially stateless between requests: like the OpenAI API, each request carries the whole conversation, and the server doesn't store chats or user data. Two things stay in memory or can be written to disk:

• Prompt cache (KV cache): --cache-prompt is on by default. It reuses the KV cache "from a previous request if possible", so a slot keeps the last prompt's state in RAM/VRAM until the next request overwrites it. Nothing is written to disk.
• Slot save/restore: only if you start the server with --slot-save-path PATH. Then POST /slots/{id}?action=save|restore|erase writes and reads KV-cache files in that directory. Leave it unset for a multi-user service unless you need it.

RAG isn't built in. You run your own retrieval: a vector store in your app, with embeddings from a separate llama-server --embeddings instance running an embedding model (/v1/embeddings), and put the retrieved text into the prompt. So the documents live in your app, not in llama-server.

Multiple users

It handles concurrent requests with slots (-np, --parallel N). Each slot is one in-flight conversation. There are no user accounts and no per-user isolation, though. If you expose it, harden it:

• Authentication: --api-key KEY (a comma-separated list) or --api-key-file. All keys have the same rights, and /health stays public. For real per-user auth, put it behind a reverse proxy that does the login, and give each user only the proxy.
• TLS: --ssl-key-file / --ssl-cert-file, or terminate TLS at the proxy.
• Turn off what users don't need: --no-slots (the /slots monitoring endpoint is enabled by default and shows per-slot state and settings), and --no-ui if you only want the API.
• Never enable the agent features on a shared server: --tools, --agent, --mcp-servers-* and --ui-mcp-proxy are all marked "do not enable in untrusted environments". --tools includes exec_shell_command, read_file and write_file on the server machine.

Configured like that, it's a reasonable shared inference backend. Keep users' data and the RAG documents in your application layer, and treat llama-server as a component that only sees one prompt at a time.

[biapar]

Thank you for your reply. I will try to understand.

[2SinceJuly]

Short answer: llama-server is not a multi-user service, and it stores nothing. Whether it is "secure" depends entirely on how you put it in front of users, because the server itself provides only two things: a shared API key and TLS.

What it does not have

• No user accounts. There is one --api-key; every caller that has it is the same principal. No per-user identity, quotas, or audit trail.
• No session isolation. Conversations are not stored server-side. /completion and /chat/completions are stateless: the client sends the full history every time. The KV cache is an optimisation keyed by slot, not a private per-user store — see below.
• No built-in RAG. Embeddings can be generated (/embedding), but there is no vector store, no retrieval, no per-user document scoping. "RAG from multiple users" has to be built outside the server entirely.

What it does have

From common/arg.cpp:

--host HOST              bind address
--port PORT
--api-key KEY            require this bearer token
--api-key-file FNAME     read the key from a file (better: not in ps/history)
--ssl-cert-file FNAME
--ssl-key-file FNAME     terminate TLS in-process

--host defaults to loopback. Binding to 0.0.0.0 without --api-key exposes an unauthenticated LLM endpoint to your network — anyone can consume your GPU. That combination is the single most common mistake, and it is what most "is it secure?" questions actually reduce to.

The KV cache and cross-user leakage

This is the part worth being precise about. The server keeps a KV cache so repeated prompts do not recompute. With default settings and multiple concurrent users, the cache holds recently processed prompt prefixes.

It does not return another user's output — there is no mechanism where user B receives user A's continuation. What it can do is retain prompt text in memory for the cache's lifetime. If you are worried about residual data in RAM, the mitigations are:

• --parallel 1 to limit concurrent slots (less interleaving of contexts)
• Restart the server between user groups
• Treat the host's memory as sensitive; it is not a hardened boundary

For a shared deployment the honest framing is: the KV cache is a performance feature, not an isolation mechanism, and you should not rely on it for tenant separation.

The architecture that actually works for multi-user

Put the server behind something that owns identity:

users → your gateway (auth, rate limit, per-user RAG, logging)
          → llama-server on 127.0.0.1:8080 with --api-key

Concretely:

1. Bind to loopback (--host 127.0.0.1), never 0.0.0.0, and put nginx/Caddy in front. TLS terminates at the proxy (simpler certificate management than --ssl-cert-file), and the proxy enforces auth.
2. One --api-key, held by the gateway only. Users never see it. Per-user keys are a gateway concept.
3. Build RAG outside. Your gateway queries its own vector store scoped by user id, assembles the context, and sends a complete prompt. llama-server never sees the documents as a corpus — only the assembled text of one request.
4. Rate limit at the gateway. There is no per-user quota in the server; a single user can saturate every slot.
5. Treat prompt logs as sensitive. If you enable any request logging, user documents end up in log files. Disable it or redact.

One more boundary

llama-server runs no user code and does not execute anything from a request; the model's output is text. The attack surface is the HTTP API (malformed requests, resource exhaustion) rather than code execution. So the risks are availability and data-in-transit/in-memory, not RCE — which is exactly why auth and binding are the two controls that matter.

[SIDDARTHAREDDY8]

Short answer: the server itself is not multi-user, so whether it is safe depends on what you put in front of it. A few things worth adding to the answers above.

There are no user accounts, but you can give each user their own key. --api-key takes a comma-separated list, and --api-key-file takes one key per line. A key is still just a shared secret though. The server has no user accounts, so it cannot tell users apart or give each one a quota. Real per-user auth and rate limits need a reverse proxy in front.

On saving data: nothing is stored by default. Conversations are stateless, the client sends the full history each time, and slot KV is only written to disk if you set --slot-save-path, which is disabled by default. What does stay in memory is the prompt cache. --cache-prompt is on by default, and the README says it reuses KV from a previous request when the prompt shares a prefix. Slots are shared across callers, so one user's cached prefix can be reused for another user's request. It does not expose the cached text through the API, but the cache is shared state between users. If that bothers you, run with --no-cache-prompt or give each user their own server instance.

On RAG: llama-server has no RAG feature. It has embedding endpoints, /v1/embeddings and /embedding, but no document store and no retrieval. Chunking, the vector store, and keeping one user's documents away from another's are all your application's job. The server just runs the model.

So the practical setup is llama-server behind a proxy that does per-user auth and rate limiting, one --api-key per user or per app, TLS with --ssl-key-file and --ssl-cert-file, and --no-cache-prompt if you care about the shared cache. If you need strict isolation between users, separate server processes per user is the simple answer.