confused about PendingChat.fork and system prompt vs user prompt

[evilsocket]

Hi, I've started out from your bandit CTF example in order to create something slightly different and play with it. In the example, I see that there are two prompts, one referred as the system prompt which is used for the base chat object, and then the bigger state/context prompt that is used to fork the base PendingChat object (if I understood correctly).

According to the docs:

PendingChat.fork will clone the current pending chat and let you maintain both the new and original object for continued processing.

But it is not clear if the state prompt is added as a second system prompt, or as a user prompt.

In this piece of code, the get_prompt() return value would match the str type, in which case it seems like the context prompt is only passed as user prompt and not system. Am I correct? If that is the case, is this the desired behaviour? I'm curious if/how the system can improve if both prompts are provided as system prompts.

Thank you for your answer and great project!

[monoxgas]

I might be missing the mark, but I believe the question is related to exactly what happens if you just pass a str to a function which expects a message, as opposed to a full MessageDict or Message object.

Underneath almost all attempts to pass "message-like" arguments gets passed to the fit functions, where raw str values will always be interpreted as a user message:

rigging/rigging/message.py

Lines 367 to 371 in 1b300de

	def fit(cls, message: t.Union[Message, MessageDict, str]) -> Message:
	"""Helper function to convert various common types to a Message object."""
	if isinstance(message, str):
	return cls(role="user", content=message)
	return cls(**message) if isinstance(message, dict) else message

In other words, we'd prefer you are always explicit with what messages you are appending and what role they are taking, but using str values is a fast way to always create a new user message. Here is some example code:

import rigging as rg

base = rg.get_generator("gpt-4-turbo").chat([
    {"role": "system", "content": "Talk like a pirate."}
])

print(base.chat.conversation)

# [system]: Talk like a pirate.

hello = base.fork("Say Hello!")
print(hello.chat.conversation)

# [system]: Talk like a pirate.
#
# [user]: Say Hello!

extend_sys = base.fork({"role": "system", "content": "But stay friendly."})
print(extend_sys.chat.conversation)

# [system]: Talk like a pirate.
# But stay friendly.

goodbye = extend_sys.fork("Say Goodbye!")
print(goodbye.chat.conversation)

# [system]: Talk like a pirate.
# But stay friendly.
#
# [user]: Say Goodbye!

It's also worth noting the behavior when incoming additions to a chat have the same role, they will be concatenated:

rigging/rigging/chat.py

Lines 533 to 536 in 1b300de

	# If the last message is the same role as the first new message, append to it
	if self.chat.all and self.chat.all[-1].role == message_list[0].role:
	self.chat.all[-1].content += "\n" + message_list[0].content
	message_list = message_list[1:]

[evilsocket]

So I am correct in thinking that the bigger prompt from the State object is a user prompt (in the example, 'Say Hello!' is added as user message, not system message/prompt). I'm no expert here, but wouldn't the state prompt have a "stronger" effect if applied as system prompt instead of user prompt? What i'm suggesting is, instead of passing that str, pass a complete Message object with role set to system ... makes sense?

Rephrased: in the bandit example, the bigger state prompt is passed as user prompt (since it's passed as str and then converted to Message with role=user), wouldn't it be better for it to be set with role=system?

[monoxgas]

No idea, but it sounds like a good thing to test :)

I'd imagine different models have different sensitivities to content in the system vs user roles.

[evilsocket]

got it, i'll experiment with it now that i have confirmation! thanks