From e4e6789be428b6fa806e723222037d0a9eeeafc2 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Thu, 5 Dec 2024 13:12:55 -0500 Subject: [PATCH 1/2] Updates LLM performance matrix (#6268) * Updates LLM performance matrix * fixes format * Updates serverless version * Excellent to great (cherry picked from commit 8e29c53736ca3faf9f805a67cacf34b47c3378d4) # Conflicts: # docs/serverless/AI-for-security/llm-performance-matrix.asciidoc --- .../llm-performance-matrix.asciidoc | 15 +++++++------- .../llm-performance-matrix.asciidoc | 20 +++++++++++++++++++ 2 files changed, 28 insertions(+), 7 deletions(-) create mode 100644 docs/serverless/AI-for-security/llm-performance-matrix.asciidoc diff --git a/docs/AI-for-security/llm-performance-matrix.asciidoc b/docs/AI-for-security/llm-performance-matrix.asciidoc index 9cf6998a87..c8f9e845c3 100644 --- a/docs/AI-for-security/llm-performance-matrix.asciidoc +++ b/docs/AI-for-security/llm-performance-matrix.asciidoc @@ -3,13 +3,14 @@ This table describes the performance of various large language models (LLMs) for different use cases in {elastic-sec}, based on our internal testing. To learn more about these use cases, refer to <> or <>. -[cols="1,1,1,1,1,1,1,1", options="header"] +[cols="1,1,1,1,1,1,1,1,1,1", options="header"] |=== -| *Feature* | *Model* | | | | | | -| | *Claude 3: Opus* | *Claude 3.5: Sonnet* | *Claude 3: Haiku* | *GPT-4o* | *GPT-4 Turbo* | **Gemini 1.5 Pro ** | **Gemini 1.5 Flash** -| *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| *Assistant - {esql} generation*| Great | Great | Poor | Excellent | Poor | Good | Poor -| *Assistant - Alert questions* | Excellent | Excellent | Excellent | Excellent | Poor | Excellent | Good -| *Attack discovery* | Excellent | Excellent | Poor | Poor | Good | Great | Poor +| *Feature* | *Model* | | | | | | | | +| | *Claude 3: Opus*| *Claude 3.5: Sonnet v2* | *Claude 3.5: Sonnet* | *Claude 3.5: Haiku*| *Claude 3: Haiku* | *GPT-4o* | *GPT-4o-mini* | **Gemini 1.5 Pro 002** | **Gemini 1.5 Flash 002** +| *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| *Assistant - {esql} generation*| Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Poor +| *Assistant - Alert questions* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Good +| *Assistant - Knowledge retrieval* | Good | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Excellent +| *Attack Discovery* | Great | Great | Excellent | Poor | Poor | Great | Poor | Excellent | Poor |=== \ No newline at end of file diff --git a/docs/serverless/AI-for-security/llm-performance-matrix.asciidoc b/docs/serverless/AI-for-security/llm-performance-matrix.asciidoc new file mode 100644 index 0000000000..193ea061ef --- /dev/null +++ b/docs/serverless/AI-for-security/llm-performance-matrix.asciidoc @@ -0,0 +1,20 @@ +[[security-llm-performance-matrix]] += Large language model performance matrix + +// :description: Learn how different models perform on different tasks in {elastic-sec}. +// :keywords: security, overview, get-started + +This table describes the performance of various large language models (LLMs) for different use cases in {elastic-sec}, based on our internal testing. To learn more about these use cases, refer to <> or <>. + + +[cols="1,1,1,1,1,1,1,1,1,1", options="header"] +|=== +| *Feature* | *Model* | | | | | | | | +| | *Claude 3: Opus*| *Claude 3.5: Sonnet v2* | *Claude 3.5: Sonnet* | *Claude 3.5: Haiku*| *Claude 3: Haiku* | *GPT-4o* | *GPT-4o-mini* | **Gemini 1.5 Pro 002** | **Gemini 1.5 Flash 002** +| *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| *Assistant - {esql} generation*| Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Poor +| *Assistant - Alert questions* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Good +| *Assistant - Knowledge retrieval* | Good | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Excellent +| *Attack Discovery* | Great | Great | Excellent | Poor | Poor | Great | Poor | Excellent | Poor +|=== + \ No newline at end of file From 5477242079312cc659801fb7bbadfe68bb60a30c Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 5 Dec 2024 18:14:28 +0000 Subject: [PATCH 2/2] Delete docs/serverless directory and its contents --- .../llm-performance-matrix.asciidoc | 20 ------------------- 1 file changed, 20 deletions(-) delete mode 100644 docs/serverless/AI-for-security/llm-performance-matrix.asciidoc diff --git a/docs/serverless/AI-for-security/llm-performance-matrix.asciidoc b/docs/serverless/AI-for-security/llm-performance-matrix.asciidoc deleted file mode 100644 index 193ea061ef..0000000000 --- a/docs/serverless/AI-for-security/llm-performance-matrix.asciidoc +++ /dev/null @@ -1,20 +0,0 @@ -[[security-llm-performance-matrix]] -= Large language model performance matrix - -// :description: Learn how different models perform on different tasks in {elastic-sec}. -// :keywords: security, overview, get-started - -This table describes the performance of various large language models (LLMs) for different use cases in {elastic-sec}, based on our internal testing. To learn more about these use cases, refer to <> or <>. - - -[cols="1,1,1,1,1,1,1,1,1,1", options="header"] -|=== -| *Feature* | *Model* | | | | | | | | -| | *Claude 3: Opus*| *Claude 3.5: Sonnet v2* | *Claude 3.5: Sonnet* | *Claude 3.5: Haiku*| *Claude 3: Haiku* | *GPT-4o* | *GPT-4o-mini* | **Gemini 1.5 Pro 002** | **Gemini 1.5 Flash 002** -| *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| *Assistant - {esql} generation*| Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Poor -| *Assistant - Alert questions* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Good -| *Assistant - Knowledge retrieval* | Good | Excellent | Excellent | Excellent | Excellent | Excellent | Great | Excellent | Excellent -| *Attack Discovery* | Great | Great | Excellent | Poor | Poor | Great | Poor | Excellent | Poor -|=== - \ No newline at end of file