Error: Unknown model for all Groq models in ChatGroq

[abishekdevendran]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import { ChatGroq } from "@langchain/groq";
const large_model = new ChatGroq({
    temperature: 0.0
});

import { ConversationSummaryBufferMemory } from "langchain/memory";
import db from "./db/sqlite";
import { small_model } from "./llms";
import {
    BaseMessage,
    StoredMessage,
    mapChatMessagesToStoredMessages,
    mapStoredMessagesToChatMessages,
} from "@langchain/core/messages";
import { BaseListChatMessageHistory } from "@langchain/core/chat_history";

// Not required, but usually chat message histories will handle multiple sessions
// for different users, and should take some kind of sessionId as input.
export interface CustomChatMessageHistoryInput {
    sessionId: string;
}

export class CustomChatMessageHistory extends BaseListChatMessageHistory {
    lc_namespace = ["langchain", "stores", "message"];

    sessionId: string;

    constructor(fields: CustomChatMessageHistoryInput) {
        super(fields);
        this.sessionId = fields.sessionId;
    }

    async getMessages(): Promise<BaseMessage[]> {
        const messagesStr = (db.prepare("SELECT data FROM chat WHERE sessionId = ?").get(this.sessionId) as {
            data: string;
        })?.data ?? "[]";
        let messages: StoredMessage[] = JSON.parse(messagesStr);
        return mapStoredMessagesToChatMessages(messages);
    }

    async addMessage(message: BaseMessage): Promise<void> {
        const retrievedMessagesStr = (db.prepare("SELECT data FROM chat WHERE sessionId = ?").get(this.sessionId) as {
            data: string;
        })?.data ?? "[]";
        let retrievedMessages: StoredMessage[] = JSON.parse(retrievedMessagesStr);
        const serializedMessages = mapChatMessagesToStoredMessages([message]);
        retrievedMessages.push(serializedMessages[0]);
        // console.log("retrievedMessage: ", retrievedMessages);
        db.prepare("INSERT OR REPLACE INTO chat (sessionId, data) VALUES (?, ?)").run(this.sessionId, JSON.stringify(retrievedMessages));
    }

    async addMessages(messages: BaseMessage[]): Promise<void> {
        const retrievedMessagesStr = (db.prepare("SELECT data FROM chat WHERE sessionId = ?").get(this.sessionId) as {
            data: string;
        })?.data ?? "[]";
        let retrievedMessages: StoredMessage[] = JSON.parse(retrievedMessagesStr);
        const serializedMessages = mapChatMessagesToStoredMessages(messages);
        retrievedMessages.push(...serializedMessages);
        // console.log("retrievedMessagessss: ", retrievedMessages);
        db.prepare("INSERT OR REPLACE INTO chat (sessionId, data) VALUES (?, ?)").run(this.sessionId, JSON.stringify(retrievedMessages));
    }

    async clear(): Promise<void> {
        db.prepare("DELETE FROM chat WHERE sessionId = ?").run(this.sessionId);
    }
}

export function getMemory(sessionId: string): ConversationSummaryBufferMemory {
    const history = new CustomChatMessageHistory({ sessionId: sessionId });
    return new ConversationSummaryBufferMemory({
        chatHistory: history,
        memoryKey: "chat_history",
        llm: small_model,
    });
}

memory.saveContext({ input: query }, { output: resp.response })

Error Message and Stack Trace (if applicable)

Failed to calculate number of tokens, falling back to approximate count Error: Unknown model
    at getEncodingNameForModel (file:///C:/Github/fc-monolith/hono/node_modules/.pnpm/js-tiktoken@1.0.12/node_modules/js-tiktoken/dist/chunk-PEBACC3C.js:230:13)
    at encodingForModel (file:///C:/Github/fc-monolith/hono/node_modules/.pnpm/@langchain+core@0.1.63_openai@4.46.1/node_modules/@langchain/core/dist/utils/tiktoken.js:19:24)
    at ChatGroq.getNumTokens (file:///C:/Github/fc-monolith/hono/node_modules/.pnpm/@langchain+core@0.1.63_openai@4.46.1/node_modules/@langchain/core/dist/language_models/base.js:177:40)
    at ConversationSummaryBufferMemory.prune (file:///C:/Github/fc-monolith/hono/node_modules/.pnpm/langchain@0.1.37_@xenova+transformers@2.17.1_@zilliz+milvus2-sdk-node@2.4.2_better-sqlite3@9._byv6wm32l75bjvaef4ttqhmd3e/node_modules/langchain/dist/memory/summary_buffer.js:114:47)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ConversationSummaryBufferMemory.saveContext (file:///C:/Github/fc-monolith/hono/node_modules/.pnpm/langchain@0.1.37_@xenova+transformers@2.17.1_@zilliz+milvus2-sdk-node@2.4.2_better-sqlite3@9._byv6wm32l75bjvaef4ttqhmd3e/node_modules/langchain/dist/memory/summary_buffer.js:96:9)

Description

I'm trying to summarize chat history and save using ConversationalSummaryBufferMemory, and when I switch out LLMs from ChatOpenAI or ChatOllama to ChatGroq, I get the error.

System Info

langchain@0.1.37 | MIT | deps: 18 | versions: 270
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.1.37.tgz
.shasum: 15db8ca5c24afc39e61773cab69e216dfb38e1bb
.integrity: sha512-rpaLEJtRrLYhAViEp7/aHfSkxbgSqHJ5n10tXv3o4kHP/wOin85RpTgewwvGjEaKc3797jOg+sLSk6a7e0UlMg==
.unpackedSize: 4.0 MB

dependencies:
@anthropic-ai/sdk: ^0.9.1 js-tiktoken: ^1.0.7 openapi-types: ^12.1.3
@langchain/community: ~0.0.47 js-yaml: ^4.1.0 p-retry: 4
@langchain/core: ~0.1.60 jsonpointer: ^5.0.1 uuid: ^9.0.0
@langchain/openai: ~0.0.28 langchainhub: ~0.0.8 yaml: ^2.2.1
@langchain/textsplitters: ~0.0.0 langsmith: ~0.1.7 zod-to-json-schema: ^3.22.3
binary-extensions: ^2.2.0 ml-distance: ^4.0.0 zod: ^3.22.4

maintainers:

• nfcampos nuno@boringbits.io
• jacoblee93 jacoblee93@gmail.com
• andrewnguonly andrewnguonly@gmail.com
• davidduong david@duong.cz
• hwchase17 hw.chase.17@gmail.com
• basproul braceasproul@gmail.com

dist-tags:
latest: 0.1.37 next: 0.2.0-rc.1

published 3 days ago by basproul braceasproul@gmail.com

@langchain/groq@0.0.9 | MIT | deps: 5 | versions: 9
Groq integration for LangChain.js
https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain-groq/

dist
.tarball: https://registry.npmjs.org/@langchain/groq/-/groq-0.0.9.tgz
.shasum: a44b19af3b784f324057bfb0217ff0613d148c2f
.integrity: sha512-/QGGgazYdxlN8FCmPfEVDO9Hg55POvQdnoou+b3lsugmwP1TYPRtqLW6JY7Atb36X4vjEJwiMCnntDXdT7vgaw==
.unpackedSize: 48.8 kB

dependencies:
@langchain/core: ~0.1.56 groq-sdk: ^0.3.2 zod: ^3.22.4
@langchain/openai: ~0.0.28 zod-to-json-schema: ^3.22.5

maintainers:

• andrewnguonly andrewnguonly@gmail.com
• basproul braceasproul@gmail.com
• jacoblee93 jacoblee93@gmail.com
• nfcampos nuno@boringbits.io
• sullivan-sean ssullivan61198@gmail.com
• hwchase17 hw.chase.17@gmail.com

dist-tags:
latest: 0.0.9

published 2 weeks ago by basproul braceasproul@gmail.com

node --version: v20.11.0
Platform: Windows 11 Version 10.0.22631 Build 22631 x64

[dosubot]

The error you're encountering with ChatGroq seems to be due to the getEncodingNameForModel function not recognizing the model. This typically happens when there's an issue with model identification, which could be due to several reasons such as an incorrect model name, a missing API key, or an improper configuration specific to ChatGroq. Here's how you can address this issue:

1. Ensure Correct Import and Instantiation: Verify that you're importing ChatGroq from @langchain/groq and instantiating it with the necessary parameters, including the apiKey. The API key is essential for the model to authenticate and function properly.

2. API Key Configuration: Double-check that the GROQ_API_KEY environment variable is correctly set. An incorrect or missing API key could lead to authentication issues, resulting in the model not being recognized.

3. Model Name and Configuration: The instantiation code you provided doesn't specify a modelName, which suggests that either the default model is being used or specifying a model name isn't required. However, ensure that your specific use case doesn't necessitate a different model name. If unsure, consulting the official ChatGroq documentation or support might be necessary.

4. Integration with ConversationSummaryBufferMemory: After ensuring the ChatGroq model is correctly instantiated, integrate it with ConversationSummaryBufferMemory as intended. Pay attention to any ChatGroq-specific configurations that might be required.

5. Debugging: If the problem persists after checking the API key and instantiation, consider debugging by verifying all parameters and configurations again. Also, keep an eye out for any updates from the Groq API that might affect model availability or parameter requirements.

By following these steps, you should be able to resolve the "Unknown model" error. If you continue to face issues, providing more details about your ChatGroq configuration and its usage context could help in further diagnosing the problem.

Details

Sources

• examples/src/memory/summary_chat.ts
• libs/langchain-groq/src/chat_models.ts
• examples/src/models/chat/chat_groq.ts
• langchain-core/src/language_models/base.ts
• langchain/src/memory/summary_buffer.ts

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[jacoblee93]

Hey, yes will have a look at this!

[jacoblee93]

Hey, apologies for the delay!

The reason this occurs is because that type of memory is dependent on token counting:

https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/memory/buffer_token_memory.ts#L115

We don't have a built-in accurate tokenizer at the moment - best we could do is add an option to suppress the warning?

[abishekdevendran]

If we just suppress the warning, will the buffer memory still continue to work to some extent? By approximating the token count somehow?

[TowhidKashem]

Hi @jacoblee93 , I'm having a similar issue as the OP except while using the ChatTogetherAI class. It seems that the memory classes only work with open ai out of the box and using any of the other providers results in the above error. The only thing I found in the docs related to token counting is this:

https://js.langchain.com/v0.1/docs/modules/model_io/chat/token_usage_tracking/

It seems to suggest that we use the token counts returned in the api response. Is that enough? Suppose i load a past conversation which is 99% close to hitting the context limit. Then the first message I send sends me over the limit. This message was already added to the context right? since there was no local way of knowing it would hit the limit, and we only found out with the response?

Even if just using the remote count is ok, at the moment the above error still occurs when using anything other than open ai. I was able to suppress it doing this:

    import llama3Tokenizer from 'llama3-tokenizer-js';
    
    ChatTogetherAI.prototype.getNumTokens = function(str) {
        const tokenCount = llama3Tokenizer.encode(str, {
          eos: false,
          bos: false
        }).length;
        return Promise.resolve(this.totalTokenCount);
    }

since I am using the llama 3 model from together ai I used this llama3Tokenizer to get the count locally. Is this enough to have the ConversationSummaryBufferMemory work and to set up local token counting? or is there more to it?

And how can I see the total count of the entire chain that the ConversationSummaryBufferMemory tracks? The above function will give me the token count for a single message but how to get the running total that the class is using to make summary and trimming decisions?

Appreciate any help! 🙏