From 0832a1dda671c64d639effca970cee9f0bfc782f Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Fri, 19 Sep 2025 13:54:42 -0400 Subject: [PATCH 1/7] Add Qwen ratings --- solutions/observability/llm-performance-matrix.md | 1 + 1 file changed, 1 insertion(+) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index f9416fa376..5228dea161 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -51,6 +51,7 @@ Models you can [deploy and manage yourself](/solutions/observability/connect-to- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Meta | **Llama-3.3-70B-Instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | | Mistral | **Mistral-Small-3.2-24B-Instruct-2506** | Excellent | Poor | Great | Great | Excellent | Poor | Good | Excellent | +| Alibaba Cloud | **Qwen2.5-72b-instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | ::::{note} `Llama-3.3-70B-Instruct` is supported with simulated function calling. From 9083d3dfc6b19c5aac18955aee7fa0f816b3997e Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 22 Oct 2025 14:19:13 -0400 Subject: [PATCH 2/7] Add Claude 4.5 to the LLM performance matrix --- solutions/observability/llm-performance-matrix.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index 5228dea161..be56b8a032 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -29,9 +29,10 @@ Models from third-party LLM providers. | Provider | Model | **Alert questions** | **APM questions** | **Contextual insights** | **Documentation retrieval** | **Elasticsearch operations** | **{{esql}} generation** | **Execute connector** | **Knowledge retrieval** | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Amazon Bedrock | **Claude Sonnet 3.5** | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Good | Excellent | -| Amazon Bedrock | **Claude Sonnet 3.7** | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Great | Excellent | +| Amazon Bedrock | **Claude Sonnet 3.5** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Good | Excellent | +| Amazon Bedrock | **Claude Sonnet 3.7** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | | Amazon Bedrock | **Claude Sonnet 4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | +| Amazon Bedrock | **Claude Sonnet 4.5** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Good | Excellent | | OpenAI | **GPT-4.1** | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Good | Excellent | | Google Gemini | **Gemini 2.0 Flash** | Excellent | Good | Excellent | Excellent | Excellent | Good | Good | Excellent | | Google Gemini | **Gemini 2.5 Flash** | Excellent | Good | Excellent | Excellent | Excellent | Good | Good | Excellent | From 1427888f7a26bd2423a32a1436b52fd9dc44415b Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 22 Oct 2025 14:28:18 -0400 Subject: [PATCH 3/7] Update Gemini ES|QL ratings --- solutions/observability/llm-performance-matrix.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index be56b8a032..13111968fc 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -35,8 +35,8 @@ Models from third-party LLM providers. | Amazon Bedrock | **Claude Sonnet 4.5** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Good | Excellent | | OpenAI | **GPT-4.1** | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Good | Excellent | | Google Gemini | **Gemini 2.0 Flash** | Excellent | Good | Excellent | Excellent | Excellent | Good | Good | Excellent | -| Google Gemini | **Gemini 2.5 Flash** | Excellent | Good | Excellent | Excellent | Excellent | Good | Good | Excellent | -| Google Gemini | **Gemini 2.5 Pro** | Excellent | Great | Excellent | Excellent | Excellent | Good | Good | Excellent | +| Google Gemini | **Gemini 2.5 Flash** | Excellent | Good | Excellent | Excellent | Excellent | Great | Good | Excellent | +| Google Gemini | **Gemini 2.5 Pro** | Excellent | Great | Excellent | Excellent | Excellent | Great | Good | Excellent | ## Open-source models [_open_source_models] From bbd1d13fe8c546779be6a9540c815ead60a41347 Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 22 Oct 2025 14:41:55 -0400 Subject: [PATCH 4/7] Add Qwen to the simulated function calling note --- solutions/observability/llm-performance-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index 13111968fc..898d616495 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -55,7 +55,7 @@ Models you can [deploy and manage yourself](/solutions/observability/connect-to- | Alibaba Cloud | **Qwen2.5-72b-instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | ::::{note} -`Llama-3.3-70B-Instruct` is supported with simulated function calling. +`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-instruct` is supported with simulated function calling. :::: ## Evaluate your own model From ab1a773e5663b034a9141b9d7f0e53b9ab463d9b Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 22 Oct 2025 14:43:38 -0400 Subject: [PATCH 5/7] Fix --- solutions/observability/llm-performance-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index 898d616495..e477dfb850 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -55,7 +55,7 @@ Models you can [deploy and manage yourself](/solutions/observability/connect-to- | Alibaba Cloud | **Qwen2.5-72b-instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | ::::{note} -`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-instruct` is supported with simulated function calling. +`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-instruct` are supported with simulated function calling. :::: ## Evaluate your own model From c4c0e76b2d5e7f8c74f1a95b4dd5b08d41753811 Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 22 Oct 2025 14:44:06 -0400 Subject: [PATCH 6/7] Fix --- solutions/observability/llm-performance-matrix.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index e477dfb850..862342600b 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -52,10 +52,10 @@ Models you can [deploy and manage yourself](/solutions/observability/connect-to- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Meta | **Llama-3.3-70B-Instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | | Mistral | **Mistral-Small-3.2-24B-Instruct-2506** | Excellent | Poor | Great | Great | Excellent | Poor | Good | Excellent | -| Alibaba Cloud | **Qwen2.5-72b-instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | +| Alibaba Cloud | **Qwen2.5-72b-Instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | ::::{note} -`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-instruct` are supported with simulated function calling. +`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-Instruct` are supported with simulated function calling. :::: ## Evaluate your own model From cb87107e14bcc1bb8147387751f55791ebfb0bd2 Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 22 Oct 2025 14:51:53 -0400 Subject: [PATCH 7/7] Update note --- solutions/observability/llm-performance-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/observability/llm-performance-matrix.md b/solutions/observability/llm-performance-matrix.md index 862342600b..7bdea946cc 100644 --- a/solutions/observability/llm-performance-matrix.md +++ b/solutions/observability/llm-performance-matrix.md @@ -55,7 +55,7 @@ Models you can [deploy and manage yourself](/solutions/observability/connect-to- | Alibaba Cloud | **Qwen2.5-72b-Instruct** | Excellent | Good | Great | Excellent | Excellent | Good | Good | Excellent | ::::{note} -`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-Instruct` are supported with simulated function calling. +`Llama-3.3-70B-Instruct` and `Qwen2.5-72b-Instruct` were tested with simulated function calling. :::: ## Evaluate your own model