Adjust mmap logic for cuda windows for faster model load #5105
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jmorganca
reviewed
Jun 17, 2024
| F16KV bool `json:"f16_kv,omitempty"` | ||
| LogitsAll bool `json:"logits_all,omitempty"` | ||
| VocabOnly bool `json:"vocab_only,omitempty"` | ||
| UseMMap TriState `json:"use_mmap,omitempty"` |
There was a problem hiding this comment.
do we need TriState here? I think false is a good default for use_mmap and then we can enable it optimistically on platforms where it will speed up loading (macOS, CPU, etc)
There was a problem hiding this comment.
The catch is we want automatic logic to toggle mmap that varies across platforms, but we have a way for the user to enable or set this, so I need to know if it's explicitly false, explicitly true, or unspecified by the user, that way the automatic logic can be disabled if the user told us exactly what they want.
jmorganca
approved these changes
Jun 17, 2024
On Windows, recent llama.cpp changes make mmap slower in most cases, so default to off. This also implements a tri-state for use_mmap so we can detect the difference between a user provided value of true/false, or unspecified.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On Windows, recent llama.cpp changes make mmap slower in most cases, so default to off. This also implements a tri-state for use_mmap so we can detect the difference between a user provided value of true/false, or unspecified.