When using RAG or Multi-Agent features with llama models the script stops replying without meeting the termination condition or errors

[gianx89]

Hi to all! I'm using Autogen to develop a RAG system, even better if Multi-Agent. I must use open source, preferably local models, like llama 3.1 or llama 3.2 . I'm using ChromaDB as my vector database.

I'm developing a system that can write a Comic Book Story with a specific format. There is writer or teams of writers that writes the story, a critic or a team of critics that gives advices on how to improve the story a manager or team of managers that incorporates this suggestions. When the story is deemed satisfactory, an agent writes "TERMINATE".

I don't have any issues using OpenAI APIs and models like GPT-3.5 Turbo or GPT-4. However, when working with open-source or local models, I encounter unpredictable behaviors.

The agents starts talking:

==============================
Starting Chat using model:  meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
==============================
Trying to create collection.
2024-11-28 16:51:13,962 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Use the existing collection `autogen-docs`.
2024-11-28 16:51:14,089 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Found 36 chunks.
VectorDB returns doc_ids:  [['45f5db71']]
Adding content of doc 45f5db71 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: Many authors began their love of storytelling by reading comics as children. Everything from The Beano and Marvel comics, to manga and graphic novels, all play a part in enriching the minds of the young and the young-at-heart.


...


--------------------------------------------------------------------------------

Next speaker: S0

Truncated 2957 tokens. Number of tokens reduced from 3366 to 409
S0 (to chat_manager):

TITLE: The Art of Superhero Storytelling
ABSTRACT: A young writer learns the art of crafting superhero comic book stories from an experienced professor.

PROTAGONISTS: 
- Professor Thompson: An experienced superhero comic writer and professor.
- Alex: A new writer who is learning to create superhero comic book stories.

PANEL START 1
 IMAGE_DESCRIPTION: Professor Thompson's office with shelves of comic books and a desk with writing materials.
 IMAGE_DIALOGUES: 
Professor Thompson: "Welcome, Alex. I see you're interested in writing superhero comics."
Alex: "Yes, Professor. I've always been a fan."
PANEL END 1

PANEL START 2
 IMAGE_DESCRIPTION: Professor Thompson holds up a comic book.
 IMAGE_DIALOGUES: 
Professor Thompson: "The key to a great superhero story is a compelling character arc."
PANEL END 2

PANEL START 3
 IMAGE_DESCRIPTION: Alex scribbles notes as Professor Thompson lectures.
 IMAGE_DIALOGUES: 
Professor Thompson: "You must balance action and dialogue to keep the reader engaged."
Alex: "That makes sense, but how do I do that?"
PANEL END 3

PANEL START 4
 IMAGE_DESCRIPTION: Professor Thompson shows Alex a split-screen of a comic panel.
 IMAGE_DIALOGUES: 
Professor Thompson: "Panel layout is crucial. It guides the reader's eye through the story."
PANEL END 4

PANEL START 5
 IMAGE_DESCRIPTION: Alex and Professor Thompson sit at a table with a large sheet of paper and pencils.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now, let's create a story together. What's your superhero's origin story?"
Alex: "Uh, I'm not sure..."
PANEL END 5

PANEL START 6
 IMAGE_DESCRIPTION: Professor Thompson smiles as Alex starts to brainstorm.
 IMAGE_DIALOGUES: 
Professor Thompson: "That's it! The origin story is the heart of any superhero tale."
PANEL END 6

PANEL START 7
 IMAGE_DESCRIPTION: Alex and Professor Thompson work on the comic script together.
 IMAGE_DIALOGUES: 
Alex: "I think I'm getting the hang of this!"
Professor Thompson: "You are! Just remember to keep it concise."
PANEL END 7

PANEL START 8
 IMAGE_DESCRIPTION: Alex and Professor Thompson review the comic script.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now, let's add some action scenes to keep the reader engaged."
Alex: "I've got an idea!"
PANEL END 8

PANEL START 9
 IMAGE_DESCRIPTION: Alex and Professor Thompson work on the comic's action scenes.
 IMAGE_DIALOGUES: 
Alex: "This is so much fun! I never realized how much work went into a comic."
Professor Thompson: "That's the beauty of it – collaboration and creativity."
PANEL END 9

PANEL START 10
 IMAGE_DESCRIPTION: Alex holds up the finished comic book, beaming with pride.
 IMAGE_DIALOGUES: 
Alex: "Thank you, Professor Thompson. I couldn't have done it without you."
Professor Thompson: "You're welcome, Alex. You're a natural."
PANEL END 10

NEXT: C0

--------------------------------------------------------------------------------

Next speaker: C0

Truncated 3204 tokens. Number of tokens reduced from 4022 to 818
C0 (to chat_manager):

NEXT: C1

--------------------------------------------------------------------------------

Next speaker: C1

Truncated 3204 tokens. Number of tokens reduced from 4026 to 822
C1 (to chat_manager):

The story is off to a good start. I like how Professor Thompson is guiding Alex through the basics of superhero storytelling.

However, I think the pacing could be improved. Some panels feel a bit rushed, such as panel 5 where Alex is asked to come up with an origin story on the spot.

I'd like to see more development of the mentor-student relationship and more exploration of the creative process.

NEXT: C0

--------------------------------------------------------------------------------

Next speaker: C0

Truncated 3204 tokens. Number of tokens reduced from 4112 to 908
C0 (to chat_manager):

I agree with C1's assessment. The story has potential, but the pacing and character development need improvement. I suggest adding more panels to show the progression of Alex's learning and the mentor-student relationship.

Specifically, I'd like to see more of Professor Thompson's teaching style and how Alex responds to it. Perhaps we could add a panel where Alex shares their own ideas and Professor Thompson provides feedback.

Additionally, let's explore the creative process more deeply. What are the challenges of writing a superhero story? How does Professor Thompson help Alex overcome these challenges?

NEXT: S0

--------------------------------------------------------------------------------

Next speaker: S0

Truncated 3204 tokens. Number of tokens reduced from 4229 to 1025
S0 (to chat_manager):

TITLE: The Art of Superhero Storytelling
ABSTRACT: A young writer learns the art of crafting superhero comic book stories from an experienced professor.

PROTAGONISTS: 
- Professor Thompson: An experienced superhero comic writer and professor.
- Alex: A new writer who is learning to create superhero comic book stories.

PANEL START 1
 IMAGE_DESCRIPTION: Professor Thompson's office with shelves of comic books and a desk with writing materials.
 IMAGE_DIALOGUES: 
Professor Thompson: "Welcome, Alex. I see you're interested in writing superhero comics."
Alex: "Yes, Professor. I've always been a fan."
PANEL END 1

PANEL START 2
 IMAGE_DESCRIPTION: Professor Thompson holds up a comic book.
 IMAGE_DIALOGUES: 
Professor Thompson: "The key to a great superhero story is a compelling character arc."
PANEL END 2

PANEL START 3
 IMAGE_DESCRIPTION: Alex scribbles notes as Professor Thompson lectures.
 IMAGE_DIALOGUES: 
Professor Thompson: "You must balance action and dialogue to keep the reader engaged."
Alex: "That makes sense, but how do I do that?"
PANEL END 3

PANEL START 4
 IMAGE_DESCRIPTION: Professor Thompson shows Alex a split-screen of a comic panel.
 IMAGE_DIALOGUES: 
Professor Thompson: "Panel layout is crucial. It guides the reader's eye through the story."
PANEL END 4

PANEL START 5
 IMAGE_DESCRIPTION: Alex and Professor Thompson sit at a table with a large sheet of paper and pencils.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now, let's brainstorm together. What kind of superhero would you like to create?"
Alex: "I've always wanted to write about a hero with the power of teleportation."
PANEL END 5

PANEL START 6
 IMAGE_DESCRIPTION: Professor Thompson smiles as Alex starts to brainstorm.
 IMAGE_DIALOGUES: 
Professor Thompson: "That's a great start! But how do you plan to make this hero relatable to readers?"
Alex: "Hmm, I'm not sure..."
PANEL END 6

PANEL START 7
 IMAGE_DESCRIPTION: Professor Thompson provides feedback as Alex shares their ideas.
 IMAGE_DIALOGUES: 
Professor Thompson: "I like where you're going with this, but what if we add a personal stake to the hero's teleportation power?"
Alex: "Like what?"
PANEL END 7

PANEL START 8
 IMAGE_DESCRIPTION: Alex's eyes light up as they start to see the potential of their hero.
 IMAGE_DIALOGUES: 
Professor Thompson: "Exactly! Now we're getting somewhere. The hero's power is not just a tool, but a reflection of their inner struggle."
Alex: "That's amazing!"
PANEL END 8

PANEL START 9
 IMAGE_DESCRIPTION: Professor Thompson hands Alex a script with notes and feedback.
 IMAGE_DIALOGUES: 
Professor Thompson: "Now it's your turn to write a scene. Don't worry about getting it perfect, just let your creativity flow."
Alex: "Thanks, Professor. I'll get right on it!"
PANEL END 9

PANEL START 10
 IMAGE_DESCRIPTION: Alex sits at their desk, writing furiously as Professor Thompson looks on with a proud smile.
 IMAGE_DIALOGUES: 
Alex: "This is it! I think I'm finally getting the hang of it!"
Professor Thompson: "That's the spirit, Alex! You're well on your way to becoming a great superhero writer."
PANEL END 10

NEXT: P0

--------------------------------------------------------------------------------

Next speaker: P0

Truncated 3520 tokens. Number of tokens reduced from 4954 to 1434
==============================
Chat Ends
==============================

Process finished with exit code 0

The chats ends without error but abruptly, without meeting the termination condition. Sometimes, very rarely, I get the right results and the chat ends with "TERMINATE" string.

Any suggestions? I can't share all the code at the moment, but I can reply with snippets of it.

P.S.
The max_rounds are pretty high (100)

[MohMaz]

@gianx89 thanks for raising the issue. Will you be able to share more how you are provisioning your local API? Anything that might help reproducing?

[gianx89]

Hi and thanks, @MohMaz.

I’ve tried the following provisioning modes:

• Ollama (local)
• Ollama with liteLLM (local)
• together.ai (remote)

I’ve been focusing on various Llama versions and tested them all in different "sizes". Here’s what I’ve tried:

• meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
• llama3.1
• llama3.2
• llama3.2-vision

I’ve also experimented with Mistral and other models in the RAG-only version of my project, but I consistently encountered problems. Now, I’m shifting my focus to the Multi-Agent aspect of the project.

I’ve tried reducing the token size (using an approximate method) and limiting the size of the chat history, but the results remain the same.

Here’s my poetry.toml file (I’m experimenting with several libraries in other packages):

[tool.poetry]
name = "comics"
version = "0.0.1"
description = "Multi Agent RAG System to write a comic book story"
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11,<3.13"
atomic-agents = "^1.0.10"
autogen = "^0.3.1"
deepeval = "^1.5.0"
instructor = "^1.0.1"
rich = "^13.9.4"
instructorembedding = "^1.0.1"
sentence-transformers = "^3.3.0"
emoji = "^2.14.0"
ipython = "^8.29.0"
markdownify = "^0.13.1"
flaml = {extras = ["automl"], version = "^2.3.2"}
pydantic = "2.9.2"
pydantic-core = "2.23.4"
ollama = "^0.3.3"
httpx = "^0.27.2"
ollama-api = "^0.1.1"
autogen-agentchat = {extras = ["together"], version = "^0.2.39"}
matplotlib = "^3.9.2"
groq = "^0.12.0"
llama-index = "^0.12.1"
llama-index-llms-ollama = "^0.4.1"
chromadb = "^0.5.20"
llama-index-vector-stores-chroma = "^0.4.0"
llama-index-embeddings-ollama = "^0.4.0"
llama-index-embeddings-huggingface = "^0.4.0"
langgraph = "^0.2.53"
pyautogen = "^0.4"
litellm = {extras = ["proxy"], version = "^1.52.16"}
together = "^1.3.5"
langchain-together = "^0.2.0"
tiktoken = "^0.8.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Reproducing the Problem

This is a link to a Python file to reproduce the problem.

Currently, it uses meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo from together.ai, but you can comment out the code to switch to using the local llama3.2-vision via Ollama.

You must provide some documents in the RAG documents folder; otherwise, you might not be able to reproduce the issue. The problem sometimes worsens when I include context from RAG.

https://www.dropbox.com/scl/fi/oslxnk78soz208wrn2gyc/autogen_multiagent_rag.py?rlkey=67u8017sn9r3fi0wat764rrsh&st=5t9lp5rm&dl=0

[ekzhu]

I think the termination condition was triggered. It could be the message transform that truncates the message and prevents the message from displaying fully. @thinkall @WaelKarkoub what do you think?

[gianx89]

It happened even without truncation. I added it trying to solve the problem. I'll post later an output without truncation. Or you can try it disabling the truncation in the provided script.

[gianx89]

Here it's an output without truncation, the same problem happens.

==============================
Starting Chat using model:  meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo
==============================
Trying to create collection.
2024-11-29 21:45:19,089 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Use the existing collection `autogen-docs`.
2024-11-29 21:45:19,243 - autogen.agentchat.contrib.retrieve_user_proxy_agent - INFO - Found 36 chunks.
VectorDB returns doc_ids:  [['45f5db71']]
Adding content of doc 45f5db71 to context.
P0 (to chat_manager):

You're a retrieve augmented chatbot. You answer user's questions based on your own knowledge and the
context provided by the user.
If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.
You must give as short an answer as possible.

User's question is: 
[INST]As P0, I will initiate the discussion. I request Team S to create a story for a comic that contains exactly 10 panels.
The story must adhere to the following guidelines:

Characters: The story should feature exactly 2 characters:
    - Character 1: An experienced superhero comic writer and professor.
    - Character 2: A new writer who is learning to create superhero comic book stories.

Story Subject:
    - The main character (professor) teaches the new writer the art of crafting superhero comic book stories.
    - The plot should reflect the dynamics of this mentor-student relationship and highlight creative and technical aspects of superhero storytelling.
Note: Ensure the story is engaging, concise, and evenly distributed across the 10 panels.

NEXT: S0.[/INST]


Context is: Many authors began their love of storytelling by reading comics as children. Everything from The Beano and Marvel 

...

NEXT: S0

--------------------------------------------------------------------------------

Next speaker: S0

S0 (to chat_manager):

I'll start creating the story based on your idea. Here's the first draft:

TITLE: The Art of Superhero Storytelling
ABSTRACT: A seasoned comic book writer and professor teaches a new writer the art of crafting superhero comic book stories.

PROTAGONISTS:
- Professor Thompson: An experienced superhero comic writer and professor.
- Alex Chen: A new writer eager to learn the art of creating superhero comic book stories.

PANEL START 1
  IMAGE_DESCRIPTION: Professor Thompson sitting at a desk with various comic books and papers scattered around. Alex Chen entering the room, looking a bit nervous.
  IMAGE_DIALOGUES:
    Professor Thompson: "Welcome to the world of superhero comics! I'm excited to share my knowledge with you."
    Alex Chen: "I've always been a huge fan of comics, but I have no idea where to start."
PANEL END 1

PANEL START 2
  IMAGE_DESCRIPTION: Close-up of Professor Thompson's face, smiling and enthusiastic.
  IMAGE_DIALOGUES:
    Professor Thompson: "Don't worry, Alex. We'll start with the basics. What makes a great superhero story?"
PANEL END 2

PANEL START 3
  IMAGE_DESCRIPTION: Alex Chen looking thoughtful, with a speech bubble above their head.
  IMAGE_DIALOGUES:
    Alex Chen: "I think it's the hero's journey. The struggle between good and evil."
PANEL END 3

PANEL START 4
  IMAGE_DESCRIPTION: Professor Thompson nodding in agreement, with a comic book panel in the background showing a classic superhero battle scene.
  IMAGE_DIALOGUES:
    Professor Thompson: "That's right! The hero's journey is a classic trope in superhero comics. But what about the characters themselves? What makes them relatable and engaging?"
PANEL END 4

PANEL START 5
  IMAGE_DESCRIPTION: Alex Chen looking puzzled, with a thought bubble above their head.
  IMAGE_DIALOGUES:
    Alex Chen: "I'm not sure. I've always just focused on the action and adventure."
PANEL END 5

PANEL START 6
  IMAGE_DESCRIPTION: Professor Thompson smiling, with a whiteboard behind them filled with notes and diagrams.
  IMAGE_DIALOGUES:
    Professor Thompson: "Ah, that's where character development comes in. Let me show you some techniques for creating well-rounded characters."
PANEL END 6

PANEL START 7
  IMAGE_DESCRIPTION: Professor Thompson drawing on the whiteboard, with Alex

--------------------------------------------------------------------------------
==============================
Chat Ends
==============================

Process finished with exit code 0

This is the link containing the code to reproduce the error.

https://www.dropbox.com/scl/fi/k27e6xp3tdp75bbefisoo/autogen_multiagent_rag_no_trunc.py?rlkey=huu8a1vc3r08h7yvmhme09l98&dl=0