Question about RLM performance on long clinical charts (local llama.cpp setup)

[Salamonti]

First, thank you for your work on RLM. I really enjoyed reading the paper. Congratulations to you and your team on a great piece of work.

I’m testing RLM locally on long clinical inputs (around 30K tokens for a patient chart), and I had a question about performance and output quality.

What I’m seeing

It takes quite a long time to process the chart compared to a normal single prompt (40 sec compared to 10 seconds)

The output seems worse for chronology and factual accuracy than a regular “single prompt” approach.

My exact setup / method

Model: mit-oasys/rlm-qwen3-8b-v0.1 converted to GGUF, served with llama.cpp’s OpenAI-compatible server at http://localhost:8081/v1.

I wrote a small FastAPI backend with a simple HTML page. The UI has two fields: “Question” and “Data/Context” (the chart text).

On submit, the backend calls your Python library like this:

rlm = RLM(backend="vllm", backend_kwargs={"base_url": "http://localhost:8081/v1", "model_name": <model_id>, "api_key": "dummy"}, verbose=False)

result = rlm.completion(prompt=context_text, root_prompt=question_text)

I display result.response in the browser.

Because llama.cpp rejects requests that exceed its context window (n_ctx=40960), I also added automatic trimming:

If llama.cpp returns “exceed_context_size_error”, I shorten the context text and retry until it fits (leaving a few thousand tokens of headroom for the RLM scaffold/system text).

Questions

Is this the right way to run “actual RLM” (prompt as context variable + root_prompt as the question) when using an OpenAI-compatible server?

Do you recommend pre-chunking a long clinical chart before calling RLM, instead of passing it as one big context variable?

Any suggestions to improve chronology and factual accuracy on medical charts (settings, prompt patterns, environment tweaks, recursion depth, etc.)?

If you think llama.cpp is a poor fit for this (vs vLLM), I’d be grateful to know that as well.

Thank you again for the excellent work.