Describe the bug
Starting from version 1.54.0 of Semantic Kernel for C#, it appears there is a regression related to the Kernel.Data property.
Up to version 1.53.1, it was possible to:

• Populate Kernel.Data before and using inside a plugin.
• Add Kernel.Data inside a plugin and using in subsequent steps.

Since version 1.54.0, any key-value pairs added inside a plugin are no longer available afterward.

Note: The Kernel instance is registered in API solution as a KeyedScoped service using dependency injection.

To Reproduce
Steps to reproduce the behavior:

• Register the Kernel as a KeyedScoped service.
• Register plugin with Plugins.AddFromObject.
• InvokeStreaming of ChatCompletionAgent with a query that trigger the plugin.
• Within a plugin add new data to Kernel.Data.
• After execution, attempt to access the data added.
• Observe that the new data is missing: only key/value pair added before invoke are present.

Expected behavior
Kernel.Data should work as a shared state across the execution pipeline.
Any data added or modified in one plugin should persist and be accessible by later steps or plugins within the same scoped instance.

Behavior was confirmed to work correctly up to v1.53.1. The issue started appearing after upgrading to v1.54.0.

Platform

• Language: C# (NET8)
• Source: NuGet package version 1.54.0
• AI model: Azure OpenAI:GPT-4o-mini(2024-07-18)
• IDE: Visual Studio 2022
• OS: Windows

Additional context
Information stored in Kernel.Data, such as retrieved document chunks, can be used after response generation to improve the quality and relevance of the knowledge stored in a vector database.

For example:

• Some documents are retrieved based on a user query (plugin input).
• documents are passed across kernel to be persisted with LLM generated response (Kernel Data).
• documents used in the generation of the response (plugin output).
• persist assistant response and retrieved document after InvokeStreaming generation

After response generation, both the original query and the model response are used to evaluate and refine the chunking and indexing strategy, improving future retrievals.