Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory limit by tokens #316

Merged
merged 8 commits into from Aug 14, 2023
Merged

Memory limit by tokens #316

merged 8 commits into from Aug 14, 2023

Conversation

javipacheco
Copy link
Contributor

@javipacheco javipacheco commented Aug 11, 2023

This PR changes the strategy for getting the messages in the conversation.

Now, we have a limit and we get the last messages for this limit. In this PR we are adding a new field to Messages, the approxTokens. In this field, we have the approximate token for every message.

When we call the memories function in the VectorStore, now we have to pass the number of token that we want to receive for the last messages in the conversation

suspend fun memories(conversationId: ConversationId, limitTokens: Int): List<Memory>

This optimises the way to get the message from the VectorStore and we are sending only the necessary tokens to OpenAI

This PR changes all of the implementations of VectorStore: LocalVectorStore, CombinedVectorStore , PGVectorStore and Lucene

@javipacheco javipacheco marked this pull request as ready for review August 11, 2023 12:15
@javipacheco
Copy link
Contributor Author

@raulraja @Montagon I have tested the xef-server on my local machine and the new approx_tokens field in xef_memory table is created. Everything works as expected

raulraja
raulraja previously approved these changes Aug 14, 2023
# Conflicts:
#	core/src/commonMain/kotlin/com/xebia/functional/xef/llm/Chat.kt
Copy link
Contributor

@raulraja raulraja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @javipacheco !

@javipacheco javipacheco merged commit 5f1130e into main Aug 14, 2023
5 checks passed
@javipacheco javipacheco deleted the memory-limit-by-tokens branch August 14, 2023 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants