Add Custom Dimensionality Support for OpenAI Embedding Components

[nstankov-bg]

Overview

This pull request introduces enhancements to the OpenAI embedding components within the Flowise repository. Specifically, new functionality has been added to the OpenAIEmbedding and OpenAIEmbeddingCustom classes allowing users to specify the dimension size of the embeddings returned by the models. This feature leverages the new capabilities of the text-embedding-3 class models from the langchain package.

Further Read.

https://js.langchain.com/docs/integrations/text_embedding/openai#specifying-dimensions

Changes

1. Addition of 'Dimensions' Parameter:

   • Both OpenAIEmbedding and OpenAIEmbeddingCustom classes now include a new input parameter labeled 'Dimensions' that accepts a number specifying the desired size of the embeddings. This parameter is optional and defaults to the maximum dimensions provided by the model if not specified.

2. Processing of 'Dimensions' Parameter:

   • The added 'dimensions' parameter is parsed and incorporated into the model configuration for embedding generation. The input is cast from a string to an integer and then passed to the model constructor, influencing the output vector's dimensionality.

3. Code Integration and Usage:

   • The dimension parameter is integrated into the existing workflows for embedding generation, ensuring that the specified dimensions are utilized when present. This integration is done without altering the existing functionalities and maintaining backward compatibility.

Justification

The necessity for this enhancement arises from the flexibility required by users dealing with various types of text data and applications. Different applications may require embeddings of different sizes to balance performance and computational efficiency. For instance, smaller dimensions might be preferred for lightweight applications or rapid prototyping, while larger dimensions might be beneficial for more complex analyses.

Impact

• Technical Impact: Developers can now customize the dimensionality of the embeddings directly from the node inputs, allowing for more flexible and application-specific text processing.
• Performance Considerations: By allowing dimension specifications, computational resources can be optimized based on the needs of specific tasks, potentially reducing memory usage and speeding up processing times for smaller dimensions.
• Usability Enhancements: This change makes the embedding components more versatile and easier to adapt to varying project requirements, enhancing the overall user experience.

Example Usage

const embeddings1024 = new OpenAIEmbeddings({
  model: "text-embedding-3-large",
  dimensions: 1024,
});

const vectors = await embeddings1024.embedDocuments(["some text"]);
console.log(vectors[0].length);  // Output: 1024

SC:

[HenryHengZJ]

That Is the most detailed PR description ever!

Btw, can you increment the version of both nodes as well? Under the this.version

And yarn lint-fix

[nstankov-bg]

Hey @HenryHengZJ, thanks for the comment! Just pushed the changes requested.

[HenryHengZJ]

Hey @HenryHengZJ, thanks for the comment! Just pushed the changes requested.

thanks!