Add Mixture-of-Miners endpoint

[dbobrenko]

Changes

• Add Mixture-of-Miners endpoint.
• Move chat completions processing and mixture of miners into two separate functions.
• Add system prompt to the inference task.

Usage example

To use mixture-of-miners, set mixture to True:

import openai
IP = "..."
PORT = "..."
API_KEY = "..."
prompt = "..."

client = openai.AsyncOpenAI(
    base_url=f"http://{IP}:{PORT}/v1",
    max_retries=0,
    api_key=API_KEY,
)

result = await client.chat.completions.create(
    model=None,
    messages=[{"role": "user", "content": prompt}],
    stream=stream,
    extra_body={
        "seed": seed,
        "sampling_parameters": {
            "temperature": 0.7,
            "top_p": 0.95,
            "top_k": 50,
            "max_new_tokens": 256,
            "do_sample": True,
            "seed": None,
        },
        "task": "QuestionAnsweringTask",
        "mixture": True,
    },
)

[Hollyqui]

See other comments

[richwardle]

LGTM

[Hollyqui]

LGTM