RFC: LatentChatAgent — Latent-space collaboration + S-LoRA for CAMEL's local model pipelines

[Arifuzzamanjoy]

Hey everyone 👋

I've been following CAMEL's work on agent scaling for a while and built something I think could be a useful addition — wanted to share it here and get the community's take before going further.

What I built: LatentMAS+S-LoRA — agents collaborate by passing latent hidden states instead of text, with role-specialized LoRA adapters (Planner, Critic, Refiner, Judger) hot-swapping on a single base model. It's already listed as a community extension in the official LatentMAS repo. (https://github.com/Gen-Verse/LatentMAS?tab=readme-ov-file#-5-latentmas-slora)

Why it's relevant to CAMEL:
CAMEL's role-playing abstraction is already the perfect semantic container for this. Instead of agents writing text messages to each other, a LatentChatAgent backend would let them pass internal representations directly — much faster, and each role gets its own specialized LoRA behavior.

Overall: 70–84% token reduction, 3–7× faster than Text-MAS.

What I'd propose:

• A LatentChatAgent class that slots into CAMEL's existing agent interface for local Qwen/LLaMA deployments
• S-LoRA adapter presets matching CAMEL's existing role types
• Optional integration with OWL's Planner/Coder/Verifier pipeline

Repo: https://github.com/Arifuzzamanjoy/latent_mas_slora
Demo: https://www.youtube.com/watch?v=g7sxYjwgRRk

Happy to hear pushback, questions, or whether this is even the direction the team wants to go. Would love to contribute properly if there's interest.

[aniruddhaadak80]

From my point of view, this is technically interesting because it attacks the token overhead in multi agent systems at the right layer.

The question I would have for CAMEL integration is observability. Once agents exchange latent states instead of text, how do you debug role behavior, compare runs, or audit why a collaboration path failed. In a research setting that is manageable, but in a framework people use for real systems, losing inspectability becomes expensive quickly.

If there is a way to pair latent exchange with optional textual summaries or checkpoints for tracing, this could fit CAMEL much more naturally.

[Arifuzzamanjoy]

Valid point -- observability is the first thing I addressed when building this.
Every agent handoff (Planner --> Critic → Refiner → Judger) passes through shared latent memory. At each boundary you can optionally project hidden states back through the language head -- single matrix multiply, not full generation. Zero cost when off.
What makes this work: each agent is tied to a named LoRA adapter, so role behavior is explicit in the weights, not implicit in a prompt. Log cosine similarity + decoded checkpoints at each handoff, and you see exactly where runs diverge.
For CAMEL: a LatentChatAgent subclass with trace_mode flag -- populates BaseMessage.content with decoded summaries when on, pushes adapter metadata into response.info. Plugs into existing logging without touching core.
Full token reduction by default, subtitles when you need them.
Repo: https://github.com/Arifuzzamanjoy/latent_mas_slora