Simple vector creation with automatic batching

Batch create vectors without thinking about it

When you're creating a lot of vectors - for example, indexing a bunch of documents at once using OpenAI embeddings - you quickly run into IO-related performance issues. Your web requests will be throttled if you make too many parallel API requests, so OpenAI allows for batched requests via the OpenAI embeddings API. However, this API only allows for a maximum of 8,191 tokens per request: about 32,764 characters.

Solution: @instant.dev/vectors provides a simple VectorManager utility that performs automatic, efficient batch creation of vectors. It will automatically collect vector creation requests over a 100ms (configurable) timeframe and batch them to minimize web requests.

It is most useful in web server contexts where multiple user requests may be creating vectors at the same time. If you rely on the same VectorManager instance all of these disparate requests will be efficiently batched.

Installation and Importing

To use this library you'll need to also work with a vector creation tool, like OpenAI.

npm i @instant.dev/vectors --save # vector management
npm i openai --save # openai for the engine

CommonJS:

const { VectorManager } = require('@instant.dev/vectors');
const OpenAI = require('openai');

const openai = new OpenAI({apiKey: process.env.OPENAI_API_KEY});
const Vectors = new VectorManager();

ESM:

import { VectorManager } from '@instant.dev/vectors';
import { Configuration, OpenAIApi } from "openai";
const configuration = new Configuration({
    organization: "YOUR_ORG_ID",
    apiKey: process.env.OPENAI_API_KEY,
});

const openai = new OpenAIApi(configuration);
const Vectors = new VectorManager();

Usage

Once you've imported and instantiated the package, it's easy to use.

Set a batch engine

// values will automatically be batched appropriately
Vectors.setEngine(async (values) => {
  const embeddingResult = await openai.embeddings.create({
    model: 'text-embedding-ada-002',
    input: values,
  });
  return embeddingResult.data.map(entry => entry.embedding);
});

Create a vector

let vector = await Vectors.create(`Something to vectorize!`);

Create multiple vectors

Manually manage vector creation:

const myStrings = [
  `Some string!`,
  `Can also be a lot longer`,
  `W`.repeat(1000),
  // ...
];

let vectors = await Promise.all(myStrings.map(str => Vectors.create(str)));

Or create multiple vectors easily with the batchCreate utility:

const myStrings = [
  `Some string!`,
  `Can also be a lot longer`,
  `W`.repeat(1000),
  // ...
];

let vectors = await Vectors.batchCreate(myStrings);

Configuration

You can configure the following parameters:

const Vectors = new VectorManager();

// these are the defaults
Vectors.maximumBatchSize = 7168 * 4; // maximum size of a batch - for OpenAI, 4 tokens per word, estimated
Vectors.maximumParallelRequests = 10; // 10 web requests simultaneously max
Vectors.fastQueueTime = 10; // time to wait if no other entries are added
Vectors.waitQueueTime = 100; // time to wait to collect entries if 1+ entries are added

Acknowledgements

Special thank you to Scott Gamble who helps run all of the front-of-house work for instant.dev 💜!

Destination	Link
Home	instant.dev
GitHub	github.com/instant-dev
Discord	discord.gg/puVYgA7ZMh
X / instant.dev	x.com/instantdevs
X / Keith Horwood	x.com/keithwhor
X / Scott Gamble	x.com/threesided