Challenges in separating "thoughts" from "content" when using langgraph4j with Spring AI 2.0.0-M4 (Streaming)

[symphony-enrico]

Hi everyone,

I’m currently working on a project using Spring AI 2.0.0-M4 with the GoogleGenAiChatModel (Gemini), and I’m trying to integrate langgraph4j to manage my agentic workflows.

In my current standalone Spring AI implementation, I leverage the streaming capabilities to provide a rich UX. Specifically, I need to distinguish between the model's "thoughts" (reasoning) and the actual "content" (final response) to display them in different UI components.

With Spring AI, this is straightforward because the Flux contains metadata that identifies the nature of the chunk:

// Example of how I currently handle it in Spring AI
if (Boolean.TRUE.equals(chatResponse.getMetadata().get("isThought"))) {
    // Render in Thought/Reasoning UI
} else {
    // Render in Chat UI
}

The Problem:
When I introduce langgraph4j as an intermediary, I process the stream via StreamingOutput (specifically NodeOutput). However, I can't find a way to access that specific metadata or distinguish between "thought" chunks and "content" chunks within the NodeOutput objects. It seems this granular metadata from the underlying Spring AI ChatResponse is lost or not propagated through the graph's streaming output.

My Questions:
Is there an existing way to access the underlying provider metadata (like isThought) within a NodeOutput when using the Spring AI integration?

If not, is this a known limitation of the current Spring AI adapter in langgraph4j?

Are there plans to expose these metadata fields in future releases to support "Reasoning" models like Gemini ?

Being able to separate reasoning from output is becoming crucial for modern LLM interfaces, so any guidance or workaround would be greatly appreciated!

Thanks in advance for the great work on this library.

Code example:

// Initialization
var stateSerializer = new SpringAIStateSerializer<>(AgentExecutorEx.State::new);
CompileConfig compileConfig = CompileConfig.builder()
        .checkpointSaver(checkpointSaver)
        .build();

  AgentExecutorEx.Builder builder = AgentExecutorEx.builder()
      .chatModel(this.chatModel)
      .defaultSystem(processedSystemPrompt)
      .stateSerializer(stateSerializer)
      .streaming(streamEnabled);

// adding some tools and mcp clients
builder.toolsFromObject(proxiedChartingToolbox);
builder.toolsFromObject(proxiedTradingView);

ToolCallback[] mcpTools = mcpClientRegistry.getToolCallbacks(agentConfig.mcpServerIds());
builder.tools(Arrays.asList(mcpTools));

return builder.build().compile(compileConfig);

[...]
// Usage
  var input = GraphInput.args(Map.of("messages", prompt.getUserMessage()));
  var runnableConfig = RunnableConfig.builder()
      .threadId("test")
      .build();

var generator = agent.stream(input, runnableConfig);

var output = generator.stream()
        .peek(s -> {
          if (s instanceof StreamingOutput<?> out) { // here I cannot distinguish thoughts/content }

[bsorrentino]

Hi @symphony-enrico thanks you for valuable feedback and for evaluating LangGraph4j

The solution is to extend NodeOutput to support metadata by implementing the appropriate HasMetadata interface.

I'll work on asap to expose all ChatResponse metadata

[symphony-enrico]

Thanks a lot @bsorrentino !

[bsorrentino]

Hi @symphony-enrico I've implemented feature and releasee it in the last 1.8-SNAPSHOT.

Please test it and let me know so I can arrange a new official release

[symphony-enrico]

Hi @bsorrentino thanks !
In quick test, I seen new metadata method, however I found it empty 😢 :

StreamingOutput{node=agent, chunk=**Reflecting on the Prompt**

I've carefully considered the user's Italian query, breaking it down into its core components. The intention is clear: the user seeks to understand my identity and capabilities. My primary goal now is to establish my role as an agent assisting a wealth manager, ensuring concise and accurate responses.


, state={
	tool_execution_requests=[]
	messages=[
	UserMessage{content='chi sei ? cosa sai fare ?', metadata={messageType=USER}, messageType=USER}
	]
},
 metadata={}
}

Usually, using Google GenAi and Gemini 3.1 Pro, Spring AI 2.0.0-M4 (I does not work with stable Spring AI) for thinking chunks like this one, I have in org.springframework.ai.chat.model.ChatResponse response :

Boolean.TRUE.equals(response.getResult().getOutput().getMetadata().get("isThought")

but it is not the case here.
Thanks

[bsorrentino]

Hi @symphony-enrico there has been a misunderstanding. When you started discussion you presented the code

// Example of how I currently handle it in Spring AI
if (Boolean.TRUE.equals(chatResponse.getMetadata().get("isThought"))) {
    // Render in Thought/Reasoning UI
} else {
    // Render in Chat UI
}

so I've propagated the ChatResponse metadata.

In the last comment you presented code

Boolean.TRUE.equals(response.getResult().getOutput().getMetadata().get("isThought")

that implies that you need AssistantMessage metadata

What metadata do you need ? Let me know

[symphony-enrico]

Hi @bsorrentino , I apologize for the confusion; you are absolutely right. The first block of code I shared was by mistake a copy-paste from an old, non-working version of my tests.

I can confirm that, in the case of GenAI, this information is indeed found in the AssistantMessage metadata. Since I couldn't find any specific documentation on this, I only discovered that the isThought flag was located there by doing some debugging and try-error.

So, yes, I do need the metadata from AssistantMessage. However, I don't think there is a real standardization yet on how to distinguish the actual response from the model's "thoughts." Switching between different AI providers (or maybe even different versions of the same provider) might mean that metadata is stored elsewhere/different naming. For this reason, I believe the best approach would be to have the ability to retrieve all metadata, while also keeping the ChatResponse metadata you have already implemented (even if I don't have a use for it right now).

Thanks a lot for your help and support!

[bsorrentino]

Hi @symphony-enrico I've worked on it and release a new proposal in last 1.8-SNAPSHOT.

For detail take a look to #374. Please test it and let me know so I can arrange a new official release

[symphony-enrico]

Hello @bsorrentino I just tested and it works great for my use case. Thanks a lot