Skip to content

Respect context size from model config #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 27, 2025
Merged

Respect context size from model config #93

merged 4 commits into from
Jun 27, 2025

Conversation

ekcasey
Copy link
Contributor

@ekcasey ekcasey commented Jun 25, 2025

Signed-off-by: Emily Casey <emily.casey@docker.com>
@ekcasey ekcasey requested a review from xenoscopic June 27, 2025 15:37
@ekcasey ekcasey marked this pull request as ready for review June 27, 2025 15:38
Comment on lines 61 to 71
if modelCfg.ContextSize != nil {
args = append(args, "--ctx-size", strconv.FormatUint(*modelCfg.ContextSize, 10))
}

// Add arguments from backend config
if config != nil {
if config.ContextSize > 0 && !containsArg(args, "--ctx-size") {
args = append(args, "--ctx-size", fmt.Sprintf("%d", config.ContextSize))
}
args = append(args, config.RuntimeFlags...)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would unify the number-to-string conversion; strconv.Itoa is probably the most efficient (vs. strconv.FormatUint or fmt.Sprintf).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can use strconv.FormatInt and strconv.FormatUint for the backend config and artifact config respectively. strconv.Itoa accepts an int type rather than an int64. I was assuming we chose int64 in the backend config for a reason and wouldn't want to risk losing precision.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, roger that.

Signed-off-by: Emily Casey <emily.casey@docker.com>
ekcasey and others added 2 commits June 27, 2025 10:47
Signed-off-by: Emily Casey <emily.casey@docker.com>

Co-authored-by: Jacob Howard <jacob.howard@docker.com>
Signed-off-by: Emily Casey <emily.casey@docker.com>
@ekcasey ekcasey requested a review from xenoscopic June 27, 2025 16:57
@ekcasey ekcasey merged commit de1ef38 into main Jun 27, 2025
4 checks passed
@ekcasey ekcasey deleted the context-size branch June 27, 2025 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants