-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments #181159
[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments #181159
Conversation
Pinging @elastic/security-solution (Team: SecuritySolution) |
…d deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](elastic#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <elastic#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below: ![langsmith_settings](https://github.com/elastic/kibana/assets/4459398/896c8155-3a06-4381-becf-b846b5aba426)
6b61234
to
f08f960
Compare
projectName?: string; | ||
runName?: string; | ||
tags?: string[]; | ||
tracers?: LangChainTracer[]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As noted when we paired, we should update this type to support our internal APMTracer
type as well so we can just add it to the array and pass along TraceOptions
wherever it needs to go. No change needed here, but something I made a note of for when I'm back working some eval changes. 👍
traceId?: string; | ||
traceOptions?: TraceOptions; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again no change necessary here, but we should look to consolidate traceId
into traceOptions
as this is getting a little confusing. The existing traceId
is intended to allow users of the ActionsClientLLM
to link any calls together so there's a single traceId
for all calls initiated from a single request from the kibana client. Again, I've made a note of this for when I work some eval changes here soon. We'll also want to standardize with the new usage of the ActionsClientChatOpenAI
over ActionsClientLLM
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checked out, code reviewed, and tested locally -- LGTM! 👍 🚀
Added a few notes around forward looking statements that we can look to address in the future, but no changes necessary here. Nice to see how little code it takes to instrument calls -- thanks for getting this improvement in @andrew-goldstein! 🙌
}: ActionsClientLlmParams) { | ||
super({}); | ||
super({ | ||
callbacks: [...(traceOptions?.tracers ?? [])], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just noting from our pairing discussions yesterday that we wanted to include the APMTracer()
here as well, but since it lives in the plugin (and ActionsClientLLM
has been moved to the common
package), we need to do a little refactoring first before adding something like:
kibana/x-pack/plugins/elastic_assistant/server/lib/langchain/executors/openai_functions_executor.ts
Line 101 in 56fe25d
const apmTracer = new APMTracer({ projectName: traceOptions?.projectName ?? 'default' }, logger); |
💚 Build Succeeded
Metrics [docs]Async chunks
History
To update your PR or re-run it, just comment with: |
…eployments (elastic#181159) ## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](elastic#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <elastic#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below: ![langsmith_settings](https://github.com/elastic/kibana/assets/4459398/896c8155-3a06-4381-becf-b846b5aba426) (cherry picked from commit 89609fe)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
…cloud deployments (#181159) (#181203) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)](#181159) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Andrew Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2024-04-18T19:01:27Z","message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n![langsmith_settings](https://github.com/elastic/kibana/assets/4459398/896c8155-3a06-4381-becf-b846b5aba426)","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Security Generative AI","v8.14.0","v8.15.0","Feature:AI Insights"],"title":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments","number":181159,"url":"#181159 Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n![langsmith_settings](#181159 Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n![langsmith_settings](https://github.com/elastic/kibana/assets/4459398/896c8155-3a06-4381-becf-b846b5aba426)","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1"}}]}] BACKPORT--> Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments
Summary
This PR enables LangSmith tracing for the AI Insights feature in cloud deployments.
LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in #180227
Details
To enable LangSmith tracing in cloud deployments, configure the following
xpack.securitySolution.enableExperimental
feature flags:assistantModelEvaluation
feature flag enables theEvaluation
settings category in the assistant. The LangSmith project name and API key are configured hereassistantAlertsInsights
feature flag enables the AI Insights feature, which is off by default at the time of this writingAfter enabling the feature flags above:
Click the
AI Assistant
button anywhere in the Security Solution to launch the assistantClick the settings gear in the assistant
Click the
Evaluation
settings categoryClick
Show Trace Options (for internal use only)
Specify a
LangSmith Project
andLangSmith API Key
per the screenshot below: