QuestionAnswerAdvisor: Augmentation of the user prompt with document context fails with structured output

[timosalm]

Bug description
The template rendering changes in M8 for the QuestionAnswerAdvisor are not working for structured output.

8:14: '"https://json-schema.org/draft/2020-12/schema"' came as a complete surprise to me

The template string is not valid.

java.lang.IllegalArgumentException: The template string is not valid.
	at org.springframework.ai.template.st.StTemplateRenderer.createST(StTemplateRenderer.java:100)
	at org.springframework.ai.template.st.StTemplateRenderer.apply(StTemplateRenderer.java:85)
	at org.springframework.ai.chat.prompt.PromptTemplate.render(PromptTemplate.java:155)
	at org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor.before(QuestionAnswerAdvisor.java:251)
	...

Probably, because the template rendering will be applied to the entire context, and fails on the JSON schema information for the structured output.

spring-ai/advisors/spring-ai-advisors-vector-store/src/main/java/org/springframework/ai/chat/client/advisor/vectorstore/QuestionAnswerAdvisor.java

Line 251 in 7bb553e

.render();

Environment
Spring AI version 1.0.0 M8

Steps to reproduce

var qaAdvisor = new QuestionAnswerAdvisor(this.vectorStore, SearchRequest.builder().build());
chatClient.prompt()
		.user(u -> u.text(userTextTemplate).param("question", "XYZ"))
		.advisors(qaAdvisor)
		.call()
		.entity(Sample.class);

Expected behavior
QuestionAnswerAdvisor works with structured output and the default system prompts provided by Spring AI

Minimal Complete Reproducible example

@ExtendWith(MockitoExtension.class)
public class QuestionAnswerAdvisorTests {

	@Mock
	ChatModel chatModel;

	@Captor
	ArgumentCaptor<Prompt> promptCaptor;

	@Captor
	ArgumentCaptor<SearchRequest> vectorSearchCaptor;

	@Mock
	VectorStore vectorStore;

	record Answer(String text) {}

	@Test
	public void qaAdvisorTakesUserTextParametersIntoAccountForSimilaritySearch() {

		given(this.chatModel.call(this.promptCaptor.capture()))
				.willReturn(new ChatResponse(List.of(new Generation(new AssistantMessage("""
				{"text":"Your answer is ZXY"}
				"""))), ChatResponseMetadata.builder().build()));

		given(this.vectorStore.similaritySearch(this.vectorSearchCaptor.capture()))
				.willReturn(List.of(new Document("doc1"), new Document("doc2")));

		var chatClient = ChatClient.builder(this.chatModel).build();
		var qaAdvisor = new QuestionAnswerAdvisor(this.vectorStore, SearchRequest.builder().build());

		var userTextTemplate = "Please answer my question {question}";
		// @formatter:off
		var answer = chatClient.prompt()
				.user(u -> u.text(userTextTemplate).param("question", "XYZ"))
				.advisors(qaAdvisor)
				.call()
				.entity(Answer.class);
		//formatter:on

		assertThat(answer).isNotNull();
		assertThat(answer.text()).isEqualTo("Your answer is ZXY");
	}

}

[g00glen00b]

This is similar to what I mentioned in #2961 (see last comment). It was suggested to use a different delimiter so that the template renderer does not fail with JSON.

[timosalm]

@ThomasVitale Is this already fixed? Setting '<' and '>' as the delimiter token for the ChatClient and Advisor works, but the structured output feature should from my perspective work in combination with the QuestionAnswerAdvisor, without any customizations.

[ilayaperumalg]

Hi @timosalm, thank you for reporting. It is a regression and @ThomasVitale has already fixed via #2963 Please use the snapshot to verify and re-open if you still see the issue.