Bug in getModelNameForTiktoken method?

[gal-checksum]

I'm using the function calculateMaxTokens to understand how many tokens I have left for a given prompt.

This function is not working properly, and upon digging further I found the issue with getModelNameForTiktoken. There's an extra "-" that are added to every startWith.

What is the reason for having a trailing dash for each model?

It creates a problem because when I pass gpt-3.5-turbo-16k as the model, it doesn't match gpt-3.5-turbo-16k- because of the extra "-" but it does match gpt-3.5-turbo, which returns the wrong model name, and thus the wrong model.

Happy to create a PR to fix, but I'm unsure if I can just remove the "-".

const getModelNameForTiktoken = (modelName) => {
    if (modelName.startsWith("gpt-3.5-turbo-16k-")) {
        return "gpt-3.5-turbo-16k";
    }
    if (modelName.startsWith("gpt-3.5-turbo-")) {
        return "gpt-3.5-turbo";
    }
    if (modelName.startsWith("gpt-4-32k-")) {
        return "gpt-4-32k";
    }
    if (modelName.startsWith("gpt-4-")) {
        return "gpt-4";
    }
    return modelName;
};

[jacoblee93]

Should be fixed by #1931 and released in 0.0.107!