This repository has been archived by the owner on Jun 24, 2024. It is now read-only.
Reserve more eval memory and use ggml scratch buffers #116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
juho-p
force-pushed
the
fix-out-of-context-memory
branch
2 times, most recently
from
1e02c17
to
b905791
Compare
philpax
reviewed
Apr 7, 2023
|
man I do not enjoy the amount of magic constants we're accumulating, but aside from the |
philpax
reviewed
Apr 7, 2023
juho-p
force-pushed
the
fix-out-of-context-memory
branch
2 times, most recently
from
248fc8c
to
bd5480c
Compare
juho-p
force-pushed
the
fix-out-of-context-memory
branch
from
bd5480c
to
d279371
Compare
|
Will this help |
No, that's because the model has a fixed context limit of 2048 tokens. We're investigating ways to improve this in #77. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This attempts to fix #115
Not the brightest of code. Scratch buffer usage is imitated from llama.cpp.
evaluatecontext memory reservation is also imitated there (or at least that's the attempt). Note that I didn't actually go through all the things in llama.cpp, only the ones related to context memory size, so I might be missing something.I'm quite sure this allocates a lot more extra memory for 7B and 13B models as well, even though I never had any issues with them running out of context memory. Scratch buffers take 1GB, and that's for every
InferenceSession.Maybe it would make sense to only use scratch buffers for inferring bigger models only? Though I think llama.cpp uses them always. Also, if there are multiple sessions, scratch memory buffers are not shared between them, but they could (just make sure only run one
evaluateat a time somehow).Anyway, with these changes I couldn't manage to run out of context memory any more, even with 65B model.