[FEATURE] Extensibility for OpenSearch Machine Learning, Phase 1 (ml-commons plugin)

[dylan-tong-aws]

What are you proposing?

We are building the first phase of extensibility capabilities for the OpenSearch ML framework. This functionality will empower ML technology providers to integrate their technology with OpenSearch via a low-to-no-code experience and join an open ecosystem that empowers builders to create AI-
powered apps faster.

Builders will be empowered to accelerate AI app development on OpenSearch through partner and community
connectors to ML technologies like cloud ML services. These connectors provide access to ML capabilities to
power solutions like semantic search, image similarity, retrieval augmented generation (RAG), personalized
search, text analytics, root cause analysis and more.

Which users have asked for this feature?

We have users within our forum that have asked for the ability to use their choice of model serving capabilities as well as the ability to more easily integrate ML APIs with our framework. There are also partners and AWS customers who have asked for the same.

What problems are you trying to solve?

1. Innovation velocity: there are so many mature and rapidly evolving model serving technologies and groundbreaking ML capabilities that are democratized exclusively through ML APIs and services. We want to let users select the best technology available to them and benefit from features that might not be natively available on OpenSearch.

2. Ease of adoption: many users have already adopted or built their own ML platform. We want to let those users leverage their existing investments and approved technologies.

3. Facilitating an open ecosystem: we need an easier way for partners and community contributors to integrate ML technologies with OpenSearch. As an open and community-driven platform, it’s important for us to empower contributors to co-innovate and drive joint-GTM motions. We want to provide integrators with a solution that ensures their engineering investments have a low cost of failure and high ROI potential.

What is the developer experience going to be?

The developer within the context of this framework is someone who is building an integration on behalf of a model
serving technology or API. The integrator creates a connector blueprint for a service like the OpenAI ChatGPT API or
Amazon SageMaker Hosting Services by defining a blueprint (eg. JSON document) that describes a protocol that
OpenSearch can use to communicate with an external ML model service.

We will share more details on the blueprint spec and APIs in the near future.

**Are there any security considerations? **

This feature will include APIs that augment the existing ml-commons APIs. The APIs will provide functionality to publish blueprints and manage connectors. We will continue to honor the same security standards by providing API-level and resource-level security controls through the OpenSearch security plugin.

This extensibility mechanism is also intended to minimize security risk for managed service providers. Unlike plugins, connectors do not require running 3rd-party code on the OpenSearch cluster. Integrators provide metadata to describes generic functionality implemented by the OpenSearch team. The metadata instructs the framework on how to interoperate with an external system through a web services based protocol.

Are there any breaking changes to the API?

No

**What is the user experience going to be? **

There are two user types: admins and end users. Admins are the users who will provision connectors using a blueprint to enable an integration with an external service endpoint to enable some feature like vector based similarity search or retrieval augmented generation (RAG) for end users. End users are the people and systems that run queries that require model inference.

This framework does not change the end user experiences that are defined by higher level features like semantic search (neural search plugin). It doesn’t effect the developers who are using existing ml-common plugin APIs. It only impacts how and where the underlying ML inference workload is executed.

The OpenSearch admins will deploy a connector designed for ML inference through the following process:

1. (Prerequisite) If the target model hasn’t already been deployed, the admin will work with MLOps experts to deploy the model on the model server technology the connector was designed to support. In the case of ML services and APIs, the models are already deployed by the service provider. The user is expected to have access to the APIs before they are able to configure and deploy a connector for a service like Amazon Comprehend, OpenAI or Cohere.

2. To deploy a community connector, the admin will download or copy the community connector definition from the OpenSearch website. The admin will use the current OpenSearch interfaces to index the template into the ml_connector index. If the admin is deploying a certified connector, they can skip this step. They can find the connector definition by searching a designated system index.

3. The admin will invoke an API to deploy the connector by specifying a blueprint and providing the inputs for the blueprint’s required parameters.

4. The OpenSearch admins can run a list command to view the active connectors.

5. The OpenSearch admin will use the features available in the OpenSearch ML Framework to monitor the external model. MLOps engineers will continue to use their incumbent ML platform and services to manage and monitor the integrated external models. These two operations work streams are decoupled. The search system is dependent on the model endpoint, but an experienced team using a ML platform or a managed services should allow continuous operations without impacting either systems.

Are there breaking changes to the User Experience?

No

Why should it be built? Any reason not to?

We’re building this feature to solve the aforementioned problems and address customer and user needs. The reason why we wouldn’t want to do this is if we wanted to keep all OpenSearch ML functionality native.

What will it take to execute?

This framework makes assumptions that a generic method of communication is possible to make no-code integrations feasible. Our research on top ML platforms, model server runtimes and APIs have validated that limiting the framework to web services (REST APIs) and a subset of ML specific commands is a practical trade-off for the benefits we gain from this approach.

Any remaining open questions?

We’re always interested in community feedback. We’re also interested in engaging with partners early to ensure we build a solution that works well with their ML offerings. Don’t hesitate to reach out to the OpenSearch Team.

[dblock]

Should we move this to ml-commons?

[dylan-tong-aws]

@dblock, yes, this should be under ml-commons. Was this added under extensions?