Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Unload model stop background #122

Merged
merged 2 commits into from
Nov 13, 2023
Merged

Unload model stop background #122

merged 2 commits into from
Nov 13, 2023

Conversation

@tikikun
Copy link
Contributor

@tikikun tikikun commented Nov 13, 2023

credit to #97 for the unloading some changes i made.

  • We will not implement a seperate function for the unload due to the fact that i might need to sync upstream from llama cpp very regularly, so direct implementation inside the background stop function will show me error everytime sync upstream
  • Add stop background process
  • Background process dependent entirely on the model_load value

@tikikun tikikun added the P0: critical Mission critical label Nov 13, 2023
@tikikun tikikun added this to the Nitro v0.2 milestone Nov 13, 2023
@tikikun tikikun self-assigned this Nov 13, 2023
This was linked to issues Nov 13, 2023
feat: Add API to unload model #86
Closed
feat: stop background processing of slots when models are stopped #118
Closed
@tikikun tikikun merged commit f49bd78 into main Nov 13, 2023
@tikikun tikikun mentioned this pull request Nov 13, 2023
Closed
@hiro-v hiro-v deleted the unload-model-stop-background branch November 13, 2023 04:46
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

@tikikun tikikun

Labels

P0: critical Mission critical

Projects

None yet

Milestone

Nitro v0.2

Development

Successfully merging this pull request may close these issues.

feat: stop background processing of slots when models are stopped feat: Add API to unload model

2 participants

@tikikun