[Bug]: 256 tokens output max

[StanGirard]

What happened?

A bug happened!

Whatever I do it is only 256 tokens output max

Relevant log output

Ask @stangirard

Twitter / LinkedIn details

No response

[dosubot]

🤖

Hello @StanGirard,

Great to see you again! I hope you're doing well.

Based on the context provided, the output limit of 256 tokens you're experiencing doesn't seem to be directly related to the Quivr codebase. The get_question_context_from_brain function in Quivr is designed to limit the total number of tokens to 3000, not 256. Here is the relevant code:

tokens = 0
for doc in documents:
    tokens += len(doc.page_content) * 1.5
    if tokens > 3000:
        documents.remove(doc)

This code is found in backend/repository/brain/get_question_context_from_brain.py.

Additionally, there is no specific function or part of the Quivr codebase that limits the output to 256 tokens according to the context provided. The function defineMaxTokens sets the maximum tokens limit based on the model used, but none of the cases in the switch statement set the limit to 256 tokens. The default case sets the limit to 1000 tokens. This function can be found in frontend/lib/helpers/defineMaxTokens.ts.

Lastly, while Quivr does not have a specific setting that limits the output to 256 tokens, there is a token limitation in the get_question_context_from_brain function where the return value of the array cannot exceed 2500 tokens, and documents are removed after reaching 2000 tokens to ensure the limit is not exceeded. This is because a token is considered to be 1.5 characters, so 2000 tokens would be equivalent to 3000 characters. This information can be found in frontend/lib/config/defaultBrainConfig.ts and backend/repository/brain/get_question_context_from_brain.py.

If the output is indeed limited to 256 tokens, it might be due to another part of the code or a different function that is not provided in the given context. Could you provide more information about the operations you're performing when you encounter this limit? This could help in identifying the source of the issue.

Sources

• backend/repository/brain/get_question_context_from_brain.py
• frontend/lib/helpers/defineMaxTokens.ts
• frontend/lib/config/defaultBrainConfig.ts