[Fleet] Consistent UX for ingest mgmt across observability and security

[mostlyjason]

Summary of the problem

As a company, we would like to offer users a consistent experience for managing ingest across the observability and security solutions so that it's easier for users to learn how to operate them and so they can adopt new solutions with a lower learning curve.

We have plans to create integrations for several additional applications including:

• Logs
• Metrics
• Uptime
• APM
• Endpoint security (already have an early working version)
• SIEM (separate from Endpoint?)

We'd like to identify what common elements of the UX should be made consistent across these applications and what customizations each will need to provide a complete experience. We'd like to coordinate this effort across teams so that we don't end up with a fractured or inconsistent UX.

There are two main modes for integrating with Ingest Manager currently:

1. Configuring integrations in Ingest Manager. The package definition allows users to specify the inputs, streams and configuration fields for each package. This acts as a form builder allowing the package author to create custom input forms for each integration.
2. Configuring integrations in the solution UI. When there are requirements not met by the custom input forms, product teams can implement their own configuration UI with their own app, and interact with Ingest management through plugins or APIs. This allows for more flexibility in the input and editing experience. When the user attempts to configure an integration in ingest manager, they are given a message to instead use the solution UI. We then need a manage directing the user between these two locations. This is how the endpoint security integration works.

For the second mode, there are several options for handling transitions between ingest manager and the solutions UIs:

1. Links direct users to the Ingest Manager app, and we save the URL to redirect users back to the solution app when they are done.
2. We offer a flyout UI that allows us to embed Ingest Manager into the solution UIs in a consistent way.
3. We offer EUI components / plugin that allow us embed Ingest Manager natively in solution UIs in a consistent way. For example, it could embed ingest management into the settings tab on the Logs UI. The downside is that it might lead to confusion around where users should go to manage ingestion since its duplicated in multiple places and they are not independent but interact with each other.

User stories

• As an admin, I would like to ingest data into the Elastic Stack across all products including Logs, Metrics, APM, Uptime, Stack monitoring, and Security with a consistent user experience.
  • I need a way to discover how to add data to the stack (such as an Add Data button).
  • I need a way to add or edit Agents and integrations, while maintaining context of what application I was working on and can return to it.

List known (technical) restrictions and requirements
None

Other
See wireframes on Whimsical, and previous POC by Endpoint team for a flyout UI.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[kevinlog]

2. We offer a flyout UI that allows us to embed Ingest Manager into the solution UIs in a consistent way.

We would like to start to migrate towards a solution like this and we're happy to test out options in our experience as we get closer to implementation. Being that this is a design ticket, we could also help frame, at least our solution, some additional use cases that may be generic enough to be options for other app.

FYI @caitlinbetz @bfishel @paul-tavares

[katrin-freihofner]

I think this is related to: https://github.com/elastic/observability-dev/issues/1122