.Net: Microsoft.SemanticKernel.Diagnostics.ModelDiagnostics.StartCompletionActivity: Collection was modified; enumeration operation may not execute.

[dimable]

Hi everyone, joining this thread since encountered the same error🙁 using .Net 9, Microsoft.SemanticKernel 1.55.0, OpenTelemetry 1.12.0.
here is the stack trace:
Collection was modified; enumeration operation may not execute.
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at Microsoft.SemanticKernel.Diagnostics.ModelDiagnostics.StartCompletionActivity[TPromptExecutionSettings](Uri endpoint, String modelName, String modelProvider, ChatHistory chatHistory, TPromptExecutionSettings executionSettings)
at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetChatMessageContentsAsync(String targetModel, ChatHistory chatHistory, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
at Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.GetChatMessageContentAsync(IChatCompletionService chatCompletionService, ChatHistory chatHistory, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
at API.AssistantApi.GetUserChatResponse(Guid id, ChatPrompt prompt, Kernel kernel, UserChatHistoryCache chatCache, CancellationToken cancellationToken) in /src/API/AssistantApi.cs:line 47
at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)
at Microsoft.AspNetCore.Http.RequestDelegateFactory.<>c__DisplayClass101_2.<b__2>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Program.<>c.<<

$>b__0_9>d.MoveNext() in /src/API/Program.cs:line 131

The error occurs when I run functional tests, just 2 consecutive requests which trigger this code in web api action:

var chatHistory = await chatCache.Get(id);
chatHistory.AddMessage(AuthorRole.User, prompt.Query);

var chat = kernel.GetRequiredService<IChatCompletionService>();
var content = await chat.GetChatMessageContentAsync(chatHistory, _promptSettings, kernel, cancellationToken);
var messageId = chatHistory.AddMessage(AuthorRole.Assistant, content.Content);
chatCache.Update(chatHistory);

return Results.Ok(new ChatResponse { Message = content.Content, ChatId = $"{chatHistory.Id}", MessageId = messageId });

When this code is executed by a call from my client app it works fine.

Originally posted by @dimable in #6058