From c39c1d1dfee9cb1d08cb157f4c5ad3dbb4667cda Mon Sep 17 00:00:00 2001 From: natasha-moore-elastic Date: Fri, 27 Jun 2025 16:07:39 +0100 Subject: [PATCH 1/3] Updates LLM performance matrix --- .../ai/large-language-model-performance-matrix.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/solutions/security/ai/large-language-model-performance-matrix.md b/solutions/security/ai/large-language-model-performance-matrix.md index ad4b764c65..af69c2fc91 100644 --- a/solutions/security/ai/large-language-model-performance-matrix.md +++ b/solutions/security/ai/large-language-model-performance-matrix.md @@ -27,16 +27,11 @@ Models from third-party LLM providers. | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | | --- | --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Claude 3.7: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude 3.5: Sonnet v2** | Excellent | Excellent | Excellent | Excellent | Great | Excellent -| | **Claude 3.5: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude 3.5: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | Poor -| | **Claude 3: Haiku** | Excellent | Excellent | Excellent | Excellent | Poor | Poor -| | **GPT-4o** | Excellent | Excellent | Excellent | Excellent | Great | Great +| **Model** | **Claude 4: Opus** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude 3.7: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude 3.5: Sonnet v4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **GPT-4o-mini** | Excellent | Great | Great | Great | Poor | Good | | **GPT-4.1** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Gemini 1.5 Pro 002** | Excellent | Excellent | Excellent | Excellent | Excellent | Great -| | **Gemini 1.5 Flash 002** | Excellent | Poor | Good | Excellent | Poor | Excellent | | **Gemini 2.0 Flash 001** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Gemini 2.5 Pro** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent @@ -47,8 +42,9 @@ Models you can [deploy yourself](/solutions/security/ai/connect-to-own-local-llm | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Mistral Nemo** | Good | Good | Great | Good | Poor | Poor | +| **Model** | **Mistral‑Small‑3.2‑24B‑Instruct‑2506** | Excellent | Poor | Excellent | Excellent | Good | N/A | | **Mistral-Small-3.1-24B-Instruct-2503** | Excellent | Poor | Excellent | Excellent | Good | N/A +| | **Mistral Nemo** | Good | Good | Great | Good | Poor | Poor | | | **LLama 3.2** | Good | Poor | Good | Poor | Poor | Good | | | **LLama 3.1 405b** | Good | Great | Good | Good | Poor | Poor | | | **LLama 3.1 70b** | Good | Good | Poor | Poor | Poor | Good | \ No newline at end of file From 147d6cf70241033940d1c4948c6121ae2e1e7226 Mon Sep 17 00:00:00 2001 From: natasha-moore-elastic Date: Mon, 14 Jul 2025 16:20:27 +0100 Subject: [PATCH 2/3] Address feedback --- .../ai/large-language-model-performance-matrix.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/solutions/security/ai/large-language-model-performance-matrix.md b/solutions/security/ai/large-language-model-performance-matrix.md index af69c2fc91..d82c4d1d04 100644 --- a/solutions/security/ai/large-language-model-performance-matrix.md +++ b/solutions/security/ai/large-language-model-performance-matrix.md @@ -27,10 +27,9 @@ Models from third-party LLM providers. | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | | --- | --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Claude 4: Opus** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude 3.7: Sonnet** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude 3.5: Sonnet v4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **GPT-4o-mini** | Excellent | Great | Great | Great | Poor | Good +| **Model** | **Claude Opus 4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude Sonnet 3.7** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude Sonnet 4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **GPT-4.1** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Gemini 2.0 Flash 001** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Gemini 2.5 Pro** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent @@ -42,8 +41,8 @@ Models you can [deploy yourself](/solutions/security/ai/connect-to-own-local-llm | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | --- | --- | --- | --- | --- | --- | --- | -| **Model** | **Mistral‑Small‑3.2‑24B‑Instruct‑2506** | Excellent | Poor | Excellent | Excellent | Good | N/A -| | **Mistral-Small-3.1-24B-Instruct-2503** | Excellent | Poor | Excellent | Excellent | Good | N/A +| **Model** | **Mistral‑Small‑3.2‑24B‑Instruct‑2506** | Excellent | Good | Excellent | Excellent | Good | N/A +| | **Mistral-Small-3.1-24B-Instruct-2503** | Excellent | Good | Excellent | Excellent | Good | N/A | | **Mistral Nemo** | Good | Good | Great | Good | Poor | Poor | | | **LLama 3.2** | Good | Poor | Good | Poor | Poor | Good | | | **LLama 3.1 405b** | Good | Great | Good | Good | Poor | Poor | From b564f9908f5d420ff4870f8c1e0bc884625dce45 Mon Sep 17 00:00:00 2001 From: natasha-moore-elastic Date: Mon, 14 Jul 2025 16:23:58 +0100 Subject: [PATCH 3/3] move chronologically --- .../security/ai/large-language-model-performance-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/security/ai/large-language-model-performance-matrix.md b/solutions/security/ai/large-language-model-performance-matrix.md index d82c4d1d04..ac85ca6b20 100644 --- a/solutions/security/ai/large-language-model-performance-matrix.md +++ b/solutions/security/ai/large-language-model-performance-matrix.md @@ -28,8 +28,8 @@ Models from third-party LLM providers. | **Feature** | - | **Assistant - General** | **Assistant - {{esql}} generation** | **Assistant - Alert questions** | **Assistant - Knowledge retrieval** | **Attack Discovery** | **Automatic Migration** | | --- | --- | --- | --- | --- | --- | --- | --- | | **Model** | **Claude Opus 4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| | **Claude Sonnet 4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Claude Sonnet 3.7** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| | **Claude Sonnet 4** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **GPT-4.1** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Gemini 2.0 Flash 001** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Gemini 2.5 Pro** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent