Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
f97dc00
SAS-994 uploading first draft
EmilyvonBergen Jan 27, 2025
80a4e54
SAS-994 Function calling documentation
EmilyvonBergen Jan 29, 2025
d0bc6a0
SAS-994 review of function calling how to
tinaengel-mx Feb 3, 2025
def45be
SAS-1112 Updating the structure of our docs
EmilyvonBergen Feb 5, 2025
3fb5863
Update _index.md
EmilyvonBergen Feb 5, 2025
978eedb
SAS-1112 Adding David doc and making last changes
EmilyvonBergen Feb 5, 2025
afe2005
Merge branch 'HowTo-FunctionCalling' of https://github.com/EmilyvonBe…
EmilyvonBergen Feb 5, 2025
a2d0bfb
Update mendix-cloud-grp.md
linaaaf Feb 5, 2025
91f84df
Update _index.md
linaaaf Feb 5, 2025
ba769a5
Merge branch 'development' of https://github.com/mendix/docs into How…
Karuna-Mendix Feb 11, 2025
d14a9b1
conflicts
Karuna-Mendix Feb 11, 2025
1aa7f3a
Proofreading
Karuna-Mendix Feb 12, 2025
566bef1
Proofreading: reference guide
Karuna-Mendix Feb 12, 2025
4f79e2c
Renamed 'function calling' to avoid duplication
Karuna-Mendix Feb 12, 2025
f4fb00a
Language review: Intrgrating Function Calling
Karuna-Mendix Feb 12, 2025
54df13c
Small fix
Karuna-Mendix Feb 12, 2025
5ef3d01
more fixes
Karuna-Mendix Feb 12, 2025
11adf7f
Merge branch 'development' of https://github.com/mendix/docs into How…
Karuna-Mendix Feb 12, 2025
1b5fdbb
Cross-references for Amazon Bedrock
Karuna-Mendix Feb 12, 2025
aa9ddaa
Cross-references: Prompt Management
Karuna-Mendix Feb 12, 2025
03d2589
Cross-references: Prompt Engineering
Karuna-Mendix Feb 12, 2025
f283773
Cross-reference: PG Vector
Karuna-Mendix Feb 12, 2025
3969cbd
Cross-reference: Mx GenAI connector
Karuna-Mendix Feb 12, 2025
266cb4c
Cross-references - GenAI Resource pack
Karuna-Mendix Feb 12, 2025
7e53c37
Cross-reference: Conversational UI
Karuna-Mendix Feb 12, 2025
2026b19
Cross-references:Open AI
Karuna-Mendix Feb 12, 2025
7b3f281
Cross-references: Vector database
Karuna-Mendix Feb 12, 2025
d149ffb
Cross-reference: Blank app
Karuna-Mendix Feb 12, 2025
215a74c
Cross-reference: Function calling
Karuna-Mendix Feb 12, 2025
5db82d8
Cross-reference: Starter app
Karuna-Mendix Feb 12, 2025
1b25637
Move Prompt Management under ref guide
Karuna-Mendix Feb 12, 2025
775de94
Change the URL for Prompt Management
Karuna-Mendix Feb 12, 2025
106b465
Cross-references: Conversational UI
Karuna-Mendix Feb 12, 2025
82f4d9b
Update links
Karuna-Mendix Feb 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ You can follow a similar approach to implement any of the other operations in **

### Chatting with Large Language Models using the ChatCompletions Operation

A common use case of the Amazon Bedrock Connector is the development of chatbots and chat solutions. The **ChatCompletions (without history / with history)** operations offer an easy way to connect to most of the text-generation models available on Amazon Bedrock. The ChatCompletions operations are built on top of Bedrock's Converse API, allowing you to talk to different models without the need of a model-specific implementation. For more information on the ChatCompletion operations, see [GenAI Commons: Chat Completions](/appstore/modules/genai/commons/#genai-generate).
A common use case of the Amazon Bedrock Connector is the development of chatbots and chat solutions. The **ChatCompletions (without history / with history)** operations offer an easy way to connect to most of the text-generation models available on Amazon Bedrock. The ChatCompletions operations are built on top of Bedrock's Converse API, allowing you to talk to different models without the need of a model-specific implementation. For more information on the ChatCompletion operations, see [GenAI Commons: Chat Completions](/appstore/modules/genai/genai-for-mx/commons/#genai-generate).

For an overview of supported models and model-specific capabilities and limitations, see [Amazon Bedrock Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html#conversation-inference-supported-models-features) in the AWS documentation.

Expand Down Expand Up @@ -187,14 +187,14 @@ To invoke a Bedrock agent for your Mendix app, do the following steps:

### Token Usage {#tokenusage}

[Token usage](/appstore/modules/genai/commons/#token-usage) monitoring is now possible for the following operations:
[Token usage](/appstore/modules/genai/genai-for-mx/commons/#token-usage) monitoring is now possible for the following operations:

* Chat Completions with History
* Chat Completion without History
* Embeddings with Cohere Embed
* Embeddings with Amazon Titan Embeddings

For more information about using this feature, refer to the [GenAI commons documentation](/appstore/modules/genai/commons/#token-usage).
For more information about using this feature, refer to the [GenAI commons documentation](/appstore/modules/genai/genai-for-mx/commons/#token-usage).

## Technical Reference {#technical-reference}

Expand All @@ -213,20 +213,20 @@ For additional information about available operations, refer to the sections bel

#### ChatCompletions (With History) and ChatCompletions (Without History) {#chat-completions}

The [ChatCompletions (with history)](/appstore/modules/genai/commons/#chat-completions-with-history) and [ChatCompletions (without history)](/appstore/modules/genai/commons/#chat-completions-without-history) activities can be used with a variety of supported LLMs.
The [ChatCompletions (with history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-with-history) and [ChatCompletions (without history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-without-history) activities can be used with a variety of supported LLMs.

Some capabilities of the chat completions operations are currently only available for specific models:

* **Function Calling** - You can use function calling in all chat completions operations using a [supported model](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html) by adding a `ToolCollection` with a `Tool` via the [Tools: Add Function to Request](/appstore/modules/genai/commons/#add-function-to-request) operation. For more information about function calling, see the [Function Calling Documentation](/appstore/modules/genai/function-calling/).
* **Function Calling** - You can use function calling in all chat completions operations using a [supported model](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html) by adding a `ToolCollection` with a `Tool` via the [Tools: Add Function to Request](/appstore/modules/genai/genai-for-mx/commons/#add-function-to-request) operation. For more information about function calling, see the [Function Calling Documentation](/appstore/modules/genai/function-calling/).

**Function calling microflows**: A microflow used as a tool for function calling must satisfy the following conditions:

1. One input parameter of type String or no input parameter.
2. Return value of type String.

* **Vision** - This operation supports the *vision* capability for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). With vision, you can send image prompts, in addition to the traditional text prompts. You can use vision by adding a `FileCollection` with a `File` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/commons/#add-file-to-collection) operation. Make sure to set the `FileType` attribute to **image**.
* **Vision** - This operation supports the *vision* capability for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). With vision, you can send image prompts, in addition to the traditional text prompts. You can use vision by adding a `FileCollection` with a `File` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/genai-for-mx/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/genai-for-mx/commons/#add-file-to-collection) operation. Make sure to set the `FileType` attribute to **image**.

* **Document Chat** - This operation supports the ability to chat with documents for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). To send a document to the model add a `FileCollection` with a `System.FileDocument` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/commons/#add-file-to-collection) operation. For Document Chat, it is not supported to create a `FileContent` from an URL using the above mentioned operations; Please use the `System.FileDocument` option. Make sure to set the `FileType` attribute to **document**.
* **Document Chat** - This operation supports the ability to chat with documents for [supported models](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html). To send a document to the model add a `FileCollection` with a `System.FileDocument` to the `Message` using the [Files: Initialize Collection with File](/appstore/modules/genai/genai-for-mx/commons/#initialize-filecollection) or the [Files: Add to Collection](/appstore/modules/genai/genai-for-mx/commons/#add-file-to-collection) operation. For Document Chat, it is not supported to create a `FileContent` from an URL using the above mentioned operations; Please use the `System.FileDocument` option. Make sure to set the `FileType` attribute to **document**.

#### RetrieveAndGenerate {#retrieve-and-generate}

Expand Down Expand Up @@ -270,25 +270,25 @@ The history can be enabled using the `SessionId` parameter on the RetrieveAndGen
This activity was introduced in Amazon Bedrock Connector version 3.1.0.
{{% /alert %}}

The [Generate Image](/appstore/modules/genai/commons/#generate-image) operation can be used to generate one or more images. Currently *Amazon Titan Image Generator G1* is the only supported model for image generation of the Amazon Bedrock Connector.
The [Generate Image](/appstore/modules/genai/genai-for-mx/commons/#generate-image) operation can be used to generate one or more images. Currently *Amazon Titan Image Generator G1* is the only supported model for image generation of the Amazon Bedrock Connector.

`GenAICommons.ImageOptions` can be an empty object. If provided, it allows you to set additional options for Image Generation and can be created by using the [Image: Create Options](/appstore/modules/genai/commons/#imageoptions-create) operation of GenAI Commons.
`GenAICommons.ImageOptions` can be an empty object. If provided, it allows you to set additional options for Image Generation and can be created by using the [Image: Create Options](/appstore/modules/genai/genai-for-mx/commons/#imageoptions-create) operation of GenAI Commons.

To retrieve actual image objects from the response, you can use the [Image: Get Generated Image (Single)](/appstore/modules/genai/commons/#image-get-single) or [Image: Get Generated Images (List)](/appstore/modules/genai/commons/#image-get-list) helper operations from GenAI Commons.
To retrieve actual image objects from the response, you can use the [Image: Get Generated Image (Single)](/appstore/modules/genai/genai-for-mx/commons/#image-get-single) or [Image: Get Generated Images (List)](/appstore/modules/genai/genai-for-mx/commons/#image-get-list) helper operations from GenAI Commons.

For Titan Image models, the `Image Generation: Add Titan Image Extension` operation can be used to configure Titan image-specific values (currently only *NegativeText*).

#### Generate Embeddings (String) {#embeddings-single-string}

The [Generate Embeddings (String)](/appstore/modules/genai/commons/#embeddings-string) activity can be used to generate an embedding vector for a given input string with one of the Cohere Embed models or Titan Embeddings v2.
The [Generate Embeddings (String)](/appstore/modules/genai/genai-for-mx/commons/#embeddings-string) activity can be used to generate an embedding vector for a given input string with one of the Cohere Embed models or Titan Embeddings v2.

For Cohere Embed and Titan Embeddings, the request can be associated to their respective EmbeddingsOptions extension object which can be created with the [Embeddings Options: Add Cohere Embed Extension](#add-cohere-embed-extension) or [Embeddings Options: Add Titan Embeddings Extension](#add-titan-embeddings-extension) operation. Through this extension, it is possible to tailor the operation to more specific needs.

Currently, embeddings are available for the Cohere Embed family and or Titan Embeddings v2.

#### Generate Embeddings (Chunk Collection) {#embeddings-chunk-collection}

The [Generate Embeddings (Chunk Collection)](/appstore/modules/genai/commons/#embeddings-chunk-collection) activity can be used to generate a collection of embedding vectors for a given collection of text chunks with one of the Cohere Embed models or Titan Embeddings v2.
The [Generate Embeddings (Chunk Collection)](/appstore/modules/genai/genai-for-mx/commons/#embeddings-chunk-collection) activity can be used to generate a collection of embedding vectors for a given collection of text chunks with one of the Cohere Embed models or Titan Embeddings v2.

For each model family, the request can be associated to an extension of the EmbeddingsOptions object which can be created with either the [Embeddings Options: Add Cohere Embed Extension](#add-cohere-embed-extension) or the [Embeddings Options: Add Titan Embeddings Extension](#add-titan-embeddings-extension) operation. Through this extension, it is possible to tailor the operation to more specific needs.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Get Started with Generative AI"
url: /appstore/modules/genai/get-started/
linktitle: "Get Started with GenAI"
weight: 10
weight: 20
description: "Describes the concepts behind generative AI and what you might implement with it."
aliases:
- /appstore/modules/genai/using-gen-ai/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ A user prompt is another fundamental type. It is the user’s input, question, o

### Context Prompt

Depending on the project or use case, adding contextual information to the model may be necessary. Normally, this information, called context prompt or conversation history, is sent in the same interaction as the system and user prompt. It captures the historical information of the conversation to maintain coherence with the end user and be context aware. In the Mendix app chatbot setup, developers configure this within their application, and it is included in the request sent to the LLM using the [Chat Completions (with history)](/appstore/modules/genai/commons/#chat-completions-with-history) operation.
Depending on the project or use case, adding contextual information to the model may be necessary. Normally, this information, called context prompt or conversation history, is sent in the same interaction as the system and user prompt. It captures the historical information of the conversation to maintain coherence with the end user and be context aware. In the Mendix app chatbot setup, developers configure this within their application, and it is included in the request sent to the LLM using the [Chat Completions (with history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-with-history) operation.

To understand this concept, imagine a user interacting with a chatbot while asking, *How should I start?*. If in previous interactions, the user asked about Mendix, the LLM will understand that the question refers to the Mendix apps. In cases where the context is not needed, such as in command-based interactions where the inquiry could be: *Turn on the lights* and the LLM does not need any historical conversation, developers can use operations like [Chat Completions (without history)](/appstore/modules/genai/commons/#chat-completions-without-history).
To understand this concept, imagine a user interacting with a chatbot while asking, *How should I start?*. If in previous interactions, the user asked about Mendix, the LLM will understand that the question refers to the Mendix apps. In cases where the context is not needed, such as in command-based interactions where the inquiry could be: *Turn on the lights* and the LLM does not need any historical conversation, developers can use operations like [Chat Completions (without history)](/appstore/modules/genai/genai-for-mx/commons/#chat-completions-without-history).

## Typical Components of a Prompt

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Every LLM will have its algorithm for generating vectors, but the convention is

#### Chunk

In the context of GenAI Commons in a Mendix app, embedding vectors are generated using a [Chunk](/appstore/modules/genai/commons/#chunk-entity). Each object represents a discrete piece of information and contains its original string representation, as well as (after the embedding operation) the vector representation of that string according to the LLM of choice.
In the context of GenAI Commons in a Mendix app, embedding vectors are generated using a [Chunk](/appstore/modules/genai/genai-for-mx/commons/#chunk-entity). Each object represents a discrete piece of information and contains its original string representation, as well as (after the embedding operation) the vector representation of that string according to the LLM of choice.

#### Knowledge base

Expand All @@ -35,11 +35,11 @@ In the context of GenAI Commons in a Mendix app, we use the [PgVector Knowledge

#### Knowledge base chunk

In most use cases, more information needs to be stored than just the original input string and its vector representation. A [KnowledgeBaseChunk](/appstore/modules/genai/commons/#knowledgebasechunk-entity) is an extension of [Chunk](/appstore/modules/genai/commons/#chunk-entity) that can hold additional information that is typically required for useful insertion and retrieval from a Mendix application.
In most use cases, more information needs to be stored than just the original input string and its vector representation. A [KnowledgeBaseChunk](/appstore/modules/genai/genai-for-mx/commons/#knowledgebasechunk-entity) is an extension of [Chunk](/appstore/modules/genai/genai-for-mx/commons/#chunk-entity) that can hold additional information that is typically required for useful insertion and retrieval from a Mendix application.

#### Metadata

If additional conventional filtering is needed during similarity searches, such additional data can be stored in the knowledge base as well. [Metadata](/appstore/modules/genai/commons/#metadata-entity) objects are key-value pairs that are inserted along with the chunks and contain this additional information. The filtering is applied on an exact string-match basis for the key-value pair. Records are only retrieved if they match all records of the metadata in the collection provided as part of the search step.
If additional conventional filtering is needed during similarity searches, such additional data can be stored in the knowledge base as well. [Metadata](/appstore/modules/genai/genai-for-mx/commons/#metadata-entity) objects are key-value pairs that are inserted along with the chunks and contain this additional information. The filtering is applied on an exact string-match basis for the key-value pair. Records are only retrieved if they match all records of the metadata in the collection provided as part of the search step.

{{% alert color="info" %}}The example described in the remainder of this document does not include the more advanced use case of metadata filtering nor does it cover the construction of complex input strings. If you want to see how this can work in practice, take a look at the *RAG with Semantic Search on Historical Data* example in the [GenAI Showcase app](https://marketplace.mendix.com/link/component/220475). {{% /alert %}}

Expand Down Expand Up @@ -69,7 +69,7 @@ In summary, in the first step, you need to provide the private knowledge base, s

Before you start experimenting with the end-to-end process, make sure that you have access to a (remote) PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension available. If you do not have one yet, [learn more](/appstore/modules/genai/pgvector-setup/) about how a PostgreSQL vector database can be set up to explore use cases with knowledge bases.

{{% alert color="info" %}}If you have access to an Amazon Web Services (AWS) account or Microsoft Azure account, Mendix recommends you use a setup described in the [Creating a PostgreSQL Database with Amazon RDS](/appstore/modules/genai/pgvector-setup/#aws-database-create) or [Managing a PostgreSQL Database with Microsoft Azure](/appstore/modules/genai/pgvector-setup/#azure-database) section. This is convenient, since these PostgreSQL databases in the cloud have the required pgvector extension available by default.{{% /alert %}}
{{% alert color="info" %}}If you have access to an Amazon Web Services (AWS) account or Microsoft Azure account, Mendix recommends you use a setup described in the [Creating a PostgreSQL Database with Amazon RDS](/appstore/modules/genai/reference-guide/external-connectors/pgvector-setup/#aws-database-create) or [Managing a PostgreSQL Database with Microsoft Azure](/appstore/modules/genai/reference-guide/external-connectors/pgvector-setup/#azure-database) section. This is convenient, since these PostgreSQL databases in the cloud have the required pgvector extension available by default.{{% /alert %}}

### Steps {#steps}

Expand Down
Loading