Tokenization and context window length

[domenic]

Models have different context window lengths and different tokenization strategies. Developers need insight into these in order to effectively work with the model.

The following are some possible ideas for useful APIs:

// Get the current state.
console.log(`
  How close are we to hitting the context window?
  ${session.tokensSoFar}/${session.maxTokens} (${session.tokensLeft} left)
`);

// Understand how large a given input prompt will be.
// Useful in combination with the above.
const numTokens = await session.countPromptTokens(promptString);

// Get notified on context window overflowing.
//
// This will by default mean the beginning of the conversation is getting lost,
// which could be problematic.
//
// (However, the system prompt will always be preserved.)
session.addEventListener("contextoverflow", e => { /* ... */ });

We're currently investigating how implementable these are.