Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

feat: support running multiple engines at the same time #891

Merged
vansangpfiev merged 3 commits into from
Jul 18, 2024
Merged

feat: support running multiple engines at the same time #891

vansangpfiev merged 3 commits into from
Jul 18, 2024

Conversation

@vansangpfiev
Copy link
Contributor

@vansangpfiev vansangpfiev commented Jul 18, 2024

Describe Your Changes

  • Currently we only allow to run one engine at a time. Because cortex.trt-llm and cortex.onnx don't support embeddings yet, therefore we need to run cortex.llamacpp for embeddings along side with cortex.trt-llm/cortex.onnx for chat completions.
  • This PR also includes a commit to refactor the codebase.

@vansangpfiev vansangpfiev requested a review from louis-jan July 18, 2024 06:54
@vansangpfiev vansangpfiev marked this pull request as ready for review July 18, 2024 06:54
louis-jan
louis-jan approved these changes Jul 18, 2024
View reviewed changes
Copy link
Contributor

@louis-jan louis-jan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vansangpfiev vansangpfiev merged commit fd7c40d into dev Jul 18, 2024
@vansangpfiev vansangpfiev deleted the feat/multi-engines branch July 18, 2024 06:59
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@louis-jan louis-jan louis-jan approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vansangpfiev @louis-jan