[Security Assistant] Adds Dataset Management API and 'Add to Dataset' Message Action

[spong]

Summary

This PR continues to address the evaluation usability enhancements outlined in https://github.com/elastic/security-team/issues/8167

As with all evaluation features, you must set the below feature flag to enable:

xpack.securitySolution.enableExperimental: [ 'assistantModelEvaluation']

This PR adds two simple REST API's for dataset management. A GET datasets route which returns an array of DataSetId's that are available for use, and a POST datasets that takes a DataSetId and a DatasetItem to persist to the dataset.

Note

Currently datasets are managed via LangSmith (and so your LangSmith API key must be configured), but this is expected to be expanded to manage datasets locally within the cluster.

Additionally, some UI elements have been updated to leverage the above endpoints:

• Dataset selection within the Evaluation UI now populates all available datasets
• A new 'Add to Dataset' message action was added enabling the ability to add a message to a dataset directly within the assistant

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)

[spong]

/ci

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: b081c05

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5451	5458	+7

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	142	143	+1
@kbn/elastic-assistant-common	217	231	+14
total			+15

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	14.6MB	14.7MB	+15.3KB

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	168	169	+1
@kbn/elastic-assistant-common	232	246	+14
total			+15

History

• 💔 Build #205076 failed 67be509

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong