Agent tool support

[xuechendi]

Support for openAI format tools and tool_choice
Request:

client.chat.completions.create(
  model=“mistral-7b",
  messages=messages,
  tools=tools,
  tool_choice="auto"
)

Reply:

{ 'role': 'assistant’, 
  'content': None, 'function_call': None, 
  'tool_calls': [{
     'id': 'call_xxxxxx’, 'type': 'function’,
     'function': {'arguments': ‘{“arg1": “..", “arg2": “.."}’, 'name’: {custom_func_name}},
}]}

Example screen shot:

Example used codes:

tools = [GetCurrentWeatherTool()]
prompt = hub.pull(args.prompt_template)
llm = ChatOpenAI(
    openai_api_base=openai_base_url,
    model_name=args.model_name,
    openai_api_key=openai_api_key,
    max_tokens=args.max_tokens,
    callbacks=[
        StreamingStdOutCallbackHandler() if args.streaming_response else StdOutCallbackHandler()
    ],
    streaming=args.streaming_response,
)
agent = create_openai_tools_agent(tools=tools, llm=llm, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({"input": "what is the weather today in Boston?"})

[xuechendi]

@carsonwang , this is initial PR to support tool calls.
The reference vLLM PR is closed not merged. This pr is following both RayLLM and vLLM.

[xuechendi]

@carsonwang , please help to review, and BTW, can you point me some one to check the right steps for adding CI functions.

[xuechendi]

Hello, @carsonwang and @jiafuzha, I'll only add http based query test in this PR. And add my langgchain/openai CI in a separate PR including both agent tool and MLLM tests.

[xuechendi]

@carsonwang , CI is added. This PR is ready for review.

[carsonwang]

@xuechendi , this is great! I will look into it.

[xuechendi]

@carsonwang , follow up on this PR, any comments?

[carsonwang]

How can other external users make this work by default? can you please add document how to launch it? If not, can you please remove and add an option to make it work for you only for now?

[xuechendi]

OK, I removed this update

[carsonwang]

This is to run the example, can we test and verify the LLM response is a correct Json to call the function?

[xuechendi]

UT verification added, only test on function name, using llama to inference

[carsonwang]

CI passed. Thanks @xuechendi !