How to calculate token size from string?

[sindresorhus]

I'm using the params.initial_prompt setting in my app, and I would like to interactively warn the user if they exceed the prompt limit. Given a string, how can I calculate the token size?

[felrock]

You could use static std::vector<whisper_vocab::id> tokenize(const whisper_vocab & vocab, const std::string & text) on your initial_prompt and check the length, compare it to your context size.

[sindresorhus]

I'm using Whisper.cpp's C API from Swift. But I now see that I could use whisper_tokenize. However, it's a bit of a waste to fill up the tokens array when I only care about the size.

So my request is to add something like this:

int whisper_token_count(struct whisper_context * ctx, const char * text) {
    const auto res = tokenize(ctx->vocab, text);
    return res.size();
}

[felrock]

Your request is a bit too specific imo. You can create a function like that yourself, just disregard the tokens after you've checked the size.

[sindresorhus]

Any app accepting a custom prompt will need this. Of course every app could reinvent the wheel, but I think many app developers don't even realize they have to validate the prompt. I for sure did not. Having a specific function like this makes it more discoverable that they need to.