When you're creating a lot of vectors - for example, indexing a bunch of documents at once using OpenAI embeddings - you quickly run into IO-related performance issues. Your web requests will be throttled if you make too many parallel API requests, so OpenAI allows for batched requests via the OpenAI embeddings API. However, this API only allows for a maximum of 8,191 tokens per request: about 32,764 characters.
Solution: @instant.dev/vectors
provides a simple VectorManager
utility that performs
automatic, efficient batch creation of vectors. It will automatically collect
vector creation requests over a 100ms (configurable) timeframe and batch them to minimize
web requests.
It is most useful in web server contexts where multiple user requests may be
creating vectors at the same time. If you rely on the same VectorManager
instance
all of these disparate requests will be efficiently batched.
To use this library you'll need to also work with a vector creation tool, like OpenAI.
npm i @instant.dev/vectors --save # vector management
npm i openai --save # openai for the engine
CommonJS:
const { VectorManager } = require('@instant.dev/vectors');
const OpenAI = require('openai');
const openai = new OpenAI({apiKey: process.env.OPENAI_API_KEY});
const Vectors = new VectorManager();
ESM:
import { VectorManager } from '@instant.dev/vectors';
import { Configuration, OpenAIApi } from "openai";
const configuration = new Configuration({
organization: "YOUR_ORG_ID",
apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);
const Vectors = new VectorManager();
Once you've imported and instantiated the package, it's easy to use.
// values will automatically be batched appropriately
Vectors.setEngine(async (values) => {
const embeddingResult = await openai.embeddings.create({
model: 'text-embedding-ada-002',
input: values,
});
return embeddingResult.data.map(entry => entry.embedding);
});
let vector = await Vectors.create(`Something to vectorize!`);
Manually manage vector creation:
const myStrings = [
`Some string!`,
`Can also be a lot longer`,
`W`.repeat(1000),
// ...
];
let vectors = await Promise.all(myStrings.map(str => Vectors.create(str)));
Or create multiple vectors easily with the batchCreate
utility:
const myStrings = [
`Some string!`,
`Can also be a lot longer`,
`W`.repeat(1000),
// ...
];
let vectors = await Vectors.batchCreate(myStrings);
You can configure the following parameters:
const Vectors = new VectorManager();
// these are the defaults
Vectors.maximumBatchSize = 7168 * 4; // maximum size of a batch - for OpenAI, 4 tokens per word, estimated
Vectors.maximumParallelRequests = 10; // 10 web requests simultaneously max
Vectors.fastQueueTime = 10; // time to wait if no other entries are added
Vectors.waitQueueTime = 100; // time to wait to collect entries if 1+ entries are added
Special thank you to Scott Gamble who helps run all of the front-of-house work for instant.dev 💜!
Destination | Link |
---|---|
Home | instant.dev |
GitHub | github.com/instant-dev |
Discord | discord.gg/puVYgA7ZMh |
X / instant.dev | x.com/instantdevs |
X / Keith Horwood | x.com/keithwhor |
X / Scott Gamble | x.com/threesided |