Add agent framework/throttling/hidden model/OS assistant and update c…

…onversational search documentation (opensearch-project#6354)

* Add agent framework documentation

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add hidden model and API updates

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Vale error

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Updated field names

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add updating credentials

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Added tools table

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add OpenSearch forum thread for OS Assistant

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add tech review for conv search

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Fix links

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add tools

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add links to tools

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* More info about tools

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Tool parameters

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Update cat-index-tool.md

Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Parameter clarification

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Tech review feedback

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Typo fix

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* More tech review feedback: RAG tool

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Tech review feedback: memory APis

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Update _ml-commons-plugin/agents-tools/index.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Update _ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Update _ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Update _ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Update _ml-commons-plugin/opensearch-assistant.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Update _ml-commons-plugin/agents-tools/tools/ppl-tool.md

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Separated search and get APIs and add conversational flow agent

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* More parameters for PPL tool

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Added more parameters

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Tech review feedback: PPL tool

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

* Rename to automating configurations

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Editorial comments on the new text

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Add parameter to PPl tool

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Changed link to configurations

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Rate limiter feedback and added warning

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

---------

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>

.github/vale/styles/Vocab/OpenSearch/Products/accept.txt

@@ -59,6 +59,8 @@ Open Distro
 OpenAI
 OpenID Connect
 OpenSearch
+OpenSearch Assistant
+OpenSearch Assistant Toolkit
 OpenSearch Benchmark
 OpenSearch Dashboards
 OpenSearch Playground

_automating-workflows/api/create-workflow.md → _automating-configurations/api/create-workflow.md

@@ -17,7 +17,7 @@ Creating a workflow adds the content of a workflow template to the flow framewor
 * Workflow step fields with invalid values.
 * Workflow graph (node/edge) configurations containing cycles or with duplicate IDs.
 
-To obtain the validation template for workflow steps, call the [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-steps/).
+To obtain the validation template for workflow steps, call the [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-steps/).
 
 Once a workflow is created, provide its `workflow_id` to other APIs.
 
@@ -50,7 +50,7 @@ POST /_plugins/_flow_framework/workflow?provision=true
 ```
 {% include copy-curl.html %}
 
-When set to `true`, the [Provision Workflow API]({{site.url}}{{site.baseurl}}/automating-workflows/api/provision-workflow/) is executed immediately following creation. 
+When set to `true`, the [Provision Workflow API]({{site.url}}{{site.baseurl}}/automating-configurations/api/provision-workflow/) is executed immediately following creation. 
 
 By default, workflows are validated when they are created to ensure that the syntax is valid and that the graph does not contain cycles. This behavior can be controlled with the `validation` query parameter. If `validation` is set to `all`, OpenSearch performs a complete template validation. Any other value of the `validation` parameter suppresses validation, allowing an incomplete/work-in-progress template to be saved. To disable template validation, set `validation` to `none`:
 

_automating-workflows/api/deprovision-workflow.md → _automating-configurations/api/deprovision-workflow.md

@@ -10,7 +10,7 @@ nav_order: 70
 This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/flow-framework/issues/475).    
 {: .warning}
 
-When you no longer need a workflow, you can deprovision its resources. Most workflow steps that create a resource have corresponding workflow steps to reverse that action. To retrieve all resources currently created for a workflow, call the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/). When you call the Deprovision Workflow API, resources included in the `resources_created` field of the Get Workflow Status API response will be removed using a workflow step corresponding to the one that provisioned them.
+When you no longer need a workflow, you can deprovision its resources. Most workflow steps that create a resource have corresponding workflow steps to reverse that action. To retrieve all resources currently created for a workflow, call the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/). When you call the Deprovision Workflow API, resources included in the `resources_created` field of the Get Workflow Status API response will be removed using a workflow step corresponding to the one that provisioned them.
 
 The workflow executes the provisioning workflow steps in reverse order. If failures occur because of resource dependencies, such as preventing deletion of a registered model if it is still deployed, the workflow attempts retries.
 
@@ -56,6 +56,6 @@ If deprovisioning did not completely remove all resources, OpenSearch responds w
 In some cases, the failure happens because of another dependent resource that took some time to be removed. In this case, you can attempt to send the same request again.
 {: .tip}
 
-To obtain a more detailed deprovisioning status than is provided by the summary in the error response, query the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/). 
+To obtain a more detailed deprovisioning status than is provided by the summary in the error response, query the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/). 
 
 On success, the workflow returns to a `NOT_STARTED` state. If some resources have not yet been removed, they are provided in the response.

_automating-workflows/api/get-workflow-status.md → _automating-configurations/api/get-workflow-status.md

@@ -10,7 +10,7 @@ nav_order: 40
 This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/flow-framework/issues/475).    
 {: .warning}
 
-[Provisioning a workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/provision-workflow/) may take a significant amount of time, particularly when the action is associated with OpenSearch indexing operations. The Get Workflow State API permits monitoring of the provisioning deployment status until it is complete.
+[Provisioning a workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/provision-workflow/) may take a significant amount of time, particularly when the action is associated with OpenSearch indexing operations. The Get Workflow State API permits monitoring of the provisioning deployment status until it is complete.
 
 ## Path and HTTP methods
 

_automating-workflows/api/get-workflow.md → _automating-configurations/api/get-workflow.md

@@ -47,4 +47,4 @@ To retrieve a template in JSON format, specify `Content-Type: application/json`
 curl -XGET "http://localhost:9200/_plugins/_flow_framework/workflow/8xL8bowB8y25Tqfenm50" -H 'Content-Type: application/json'
 ```
 
-OpenSearch responds with the stored template containing the same content as the body of the [create workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/create-workflow/) request. The order of fields in the returned template may not exactly match the original template but will function identically.
+OpenSearch responds with the stored template containing the same content as the body of the [create workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/) request. The order of fields in the returned template may not exactly match the original template but will function identically.

_automating-workflows/api/index.md → _automating-configurations/api/index.md

@@ -13,11 +13,11 @@ This is an experimental feature and is not recommended for use in a production e
 
 OpenSearch supports the following workflow APIs:
 
-* [Create or update workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/create-workflow/)
-* [Get workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow/)
-* [Provision workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/provision-workflow/)
-* [Get workflow status]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/)
-* [Get workflow steps]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-steps/)
-* [Search workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/search-workflow/)
-* [Deprovision workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/deprovision-workflow/)
-* [Delete workflow]({{site.url}}{{site.baseurl}}/automating-workflows/api/delete-workflow/)
+* [Create or update workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/)
+* [Get workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow/)
+* [Provision workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/provision-workflow/)
+* [Get workflow status]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/)
+* [Get workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-steps/)
+* [Search workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/search-workflow/)
+* [Deprovision workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/deprovision-workflow/)
+* [Delete workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/delete-workflow/)

_automating-workflows/api/provision-workflow.md → _automating-configurations/api/provision-workflow.md

@@ -12,7 +12,7 @@ This is an experimental feature and is not recommended for use in a production e
 
 Provisioning a workflow is a one-time setup process usually performed by a cluster administrator to create resources that will be used by end users.  
 
-The `workflows` template field may contain multiple workflows. The workflow with the `provision` key can be executed with this API. This API is also executed when the [Create or Update Workflow API]({{site.url}}{{site.baseurl}}/automating-workflows/api/create-workflow/) is called with the `provision` parameter set to `true`.
+The `workflows` template field may contain multiple workflows. The workflow with the `provision` key can be executed with this API. This API is also executed when the [Create or Update Workflow API]({{site.url}}{{site.baseurl}}/automating-configurations/api/create-workflow/) is called with the `provision` parameter set to `true`.
 
 You can only provision a workflow if it has not yet been provisioned. Deprovision the workflow if you need to repeat provisioning.
 {: .note}
@@ -48,4 +48,4 @@ OpenSearch responds with the same `workflow_id` that was used in the request:
 }
 ```
 
-To obtain the provisioning status, query the [Get Workflow State API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-status/).
+To obtain the provisioning status, query the [Get Workflow State API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/).

_automating-workflows/index.md → _automating-configurations/index.md

@@ -1,13 +1,13 @@
 ---
 layout: default
-title: Automating workflows
+title: Automating configurations
 nav_order: 1
 has_children: false
 nav_exclude: true
-redirect_from: /automating-workflows/
+redirect_from: /automating-configurations/
 ---
 
-# Automating workflows
+# Automating configurations
 **Introduced 2.12**
 {: .label .label-purple }
 
@@ -16,7 +16,7 @@ This is an experimental feature and is not recommended for use in a production e
 
 You can automate complex OpenSearch setup and preprocessing tasks by providing templates for common use cases. For example, automating machine learning (ML) setup tasks streamlines the use of OpenSearch ML offerings.
 
-In OpenSearch 2.12, workflow automation is limited to ML tasks.
+In OpenSearch 2.12, configuration automation is limited to ML tasks.
 {: .info}
 
 OpenSearch use case templates provide a compact description of the setup process in a JSON or YAML document. These templates describe automated workflow configurations for conversational chat or query generation, AI connectors, tools, agents, and other components that prepare OpenSearch as a backend for generative models. For template examples, see [Sample templates](https://github.com/opensearch-project/flow-framework/tree/main/sample-templates).
@@ -38,12 +38,12 @@ Workflow automation provides the following benefits:
 * **Workflows**: One or more workflows containing the following elements:
     * **User input**: Parameters expected from the user that are specific to the steps in this workflow.
     * **Workflow Steps**: The workflow steps described as a directed acyclic graph (DAG):  
-        * ***Nodes*** describe steps of the process, which may be executed in parallel. For the syntax of workflow steps, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-steps/). 
+        * ***Nodes*** describe steps of the process, which may be executed in parallel. For the syntax of workflow steps, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-steps/). 
         * ***Edges*** sequence nodes to be executed after the previous step is complete and may use the output fields of previous steps. When a node includes a key in the `previous_node_input` map referring to a previous node’s workflow step, a corresponding edge is automatically added to the template during parsing and may be omitted for the sake of simplicity.
 
 ## Next steps
 
-- For supported APIs, see [Workflow APIs]({{site.url}}{{site.baseurl}}/automating-workflows/api/index/).
-- For the workflow step syntax, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-steps/).  
-- For a complete example, see [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-tutorial/).
-- For configurable settings, see [Workflow settings]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-settings/).
+- For supported APIs, see [Workflow APIs]({{site.url}}{{site.baseurl}}/automating-configurations/api/index/).
+- For the workflow step syntax, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-steps/).  
+- For a complete example, see [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/).
+- For configurable settings, see [Workflow settings]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-settings/).

_automating-workflows/workflow-steps.md → _automating-configurations/workflow-steps.md

@@ -20,7 +20,7 @@ Workflow steps are actively being developed to expand automation capabilities. W
 |`id`	|String	|Required	| A user-provided ID for the step. The ID must be unique within a given workflow and is useful for identifying resources created by the step. For example, a `register_agent` step may return an `agent_id` that has been registered. Using this ID, you can determine which step produced which resource.	|
 |`type`	|String	|Required	|The type of action to take, such as `deploy_model`, which corresponds to the API for which the step is used. Multiple steps may share the same type but must each have their own unique ID. For a list of supported types, see [Workflow step types](#workflow-step-types).	|
 |`previous_node_inputs`	|Object	|Optional	| A key-value map specifying user inputs that are produced by a previous step in the workflow. For each key-value pair, the key is the previous step's `id` and the value is an API body field name (such as `model_id`) that will be produced as an output of a previous step in the workflow. For example, `register_remote_model` (key) may produce a `model_id` (value) that is required for a subsequent `deploy_model` step. <br> A graph edge is automatically added to the workflow connecting the previous step's key as the source and the current node as the destination. <br>In some cases, you can include [additional inputs](#additional-fields) in this field.	|
-|`user_inputs`	|Object	|Optional	| A key-value map of inputs supported by the corresponding API for this specific step. Some inputs are required for an API, while others are optional. Required inputs may be specified here, if known, or in the `previous_node_inputs` field. The [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-workflows/api/get-workflow-steps/) identifies required inputs and step outputs. <br> Substitutions are supported in string values, lists of strings, and maps with string values. The pattern `{% raw %}${{previous_step_id.output_key}}{% endraw %}` will be replaced by the value in the previous step's output with the given key.  For example, if a parameter map in the user inputs includes a key `embedding_model_id` with a value `{% raw %}${{deploy_embedding_model.model_id}}{% endraw %}`, then the `model_id` output of the `deploy_embedding_model` step will be substituted here. This performs a similar function to the `previous_node_input` map but is not validated and does not automatically infer edges. <br>In some cases, you can include [additional inputs](#additional-fields) in this field.	|
+|`user_inputs`	|Object	|Optional	| A key-value map of inputs supported by the corresponding API for this specific step. Some inputs are required for an API, while others are optional. Required inputs may be specified here, if known, or in the `previous_node_inputs` field. The [Get Workflow Steps API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-steps/) identifies required inputs and step outputs. <br> Substitutions are supported in string values, lists of strings, and maps with string values. The pattern `{% raw %}${{previous_step_id.output_key}}{% endraw %}` will be replaced by the value in the previous step's output with the given key.  For example, if a parameter map in the user inputs includes a key `embedding_model_id` with a value `{% raw %}${{deploy_embedding_model.model_id}}{% endraw %}`, then the `model_id` output of the `deploy_embedding_model` step will be substituted here. This performs a similar function to the `previous_node_input` map but is not validated and does not automatically infer edges. <br>In some cases, you can include [additional inputs](#additional-fields) in this field.	|
 
 ## Workflow step types
 
@@ -61,4 +61,4 @@ You can include the following additional fields in the `previous_node_inputs` fi
 
 ## Example workflow steps
 
-For example workflow step implementations, see the [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-workflows/workflow-tutorial/).
+For example workflow step implementations, see the [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/).

_config.yml

@@ -112,7 +112,7 @@ collections:
   about:
     permalink: /:collection/:path/
     output: true
-  automating-workflows:
+  automating-configurations:
     permalink: /:collection/:path/
     output: true
   dashboards-assistant:
@@ -172,11 +172,8 @@ opensearch_collection:
     ml-commons-plugin:
       name: Machine learning
       nav_fold: true
-    dashboards-assistant:
-      name: Dashboards assistant
-      nav_fold: true
-    automating-workflows:
-      name: Automating workflows
+    automating-configurations:
+      name: Automating configurations
       nav_fold: true
     monitoring-your-cluster:
       name: Monitoring your cluster