Skip to content

[Feature]: Every node hold a local chatbot #106

@gufengc

Description

@gufengc

Is your feature request related to a problem?

Current only scheduler node can hold chatbot, we should let every node can hold a local chatbot.

Describe the Solution you'd like

  1. Reuse current chatbot frontend
  2. Build a new simple http server for node, http server forward request(chat completion and cluster status api) to scheduler lattica rpc handler @sibianl
  3. scheduler rpc handler forward request to scheduler http api @gufengc
  4. launch frontend and http server on node(do not launch it on scheduler) @sibianl

Alternatives Considered (Optional)

No response

Additional Context (Optional)

No response

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions