[Refactor] support more file/array schema in tools

[wj-Mcat]

No description provided.

[wj-Mcat]

在 APIHub 上面预支 tool 的执行结果：

============================= test session starts ==============================
platform darwin -- Python 3.10.13, pytest-7.4.2, pluggy-1.3.0 -- /Users/wujingjing05/miniconda3/envs/eb-agent/bin/python
cachedir: .pytest_cache
rootdir: /Users/wujingjing05/projects/yiyan/ERNIE-Bot-SDK-add-tool-unittest/erniebot-agent
plugins: cov-4.1.0, asyncio-0.23.2, anyio-4.2.0
asyncio: mode=strict
collecting ... collected 24 items

tests/integration_tests/apihub/test_car_cls.py::TestRemoteTool::test_car_classify PASSED [  4%]
tests/integration_tests/apihub/test_dish.py::TestRemoteTool::test_dish_classify PASSED [  8%]
tests/integration_tests/apihub/test_doc_rm_bnd.py::TestRemoteTool::test_doc_rm_bnd PASSED [ 12%]
tests/integration_tests/apihub/test_handwriting.py::TestRemoteTool::test_hand_text_rec PASSED [ 16%]
tests/integration_tests/apihub/test_high_ocr.py::TestRemoteTool::test_tool PASSED [ 20%]
tests/integration_tests/apihub/test_image_moderation.py::TestRemoteTool::test_tool PASSED [ 25%]
tests/integration_tests/apihub/test_img_enhance.py::TestRemoteTool::test_tool PASSED [ 29%]
tests/integration_tests/apihub/test_img_transform.py::TestRemoteTool::test_img_style_trans PASSED [ 33%]
tests/integration_tests/apihub/test_img_transform.py::TestRemoteTool::test_person_animation PASSED [ 37%]
tests/integration_tests/apihub/test_landmark.py::TestRemoteTool::test_dish_classify PASSED [ 41%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_formula FAILED [ 45%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_ocr_general PASSED [ 50%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_ocr_pp PASSED [ 54%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_pp_ocr_v4 PASSED [ 58%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_shopping_receipt FAILED [ 62%]
tests/integration_tests/apihub/test_pic_translate.py::TestRemoteTool::test_tool FAILED [ 66%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_matting PASSED [ 70%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_ocr_v4 PASSED [ 75%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_structure PASSED [ 79%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_tinypose PASSED [ 83%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_vehicle PASSED [ 87%]
tests/integration_tests/apihub/test_text_moderation.py::TestRemoteTool::test_tool PASSED [ 91%]
tests/integration_tests/apihub/test_text_to_speech.py::TestRemoteTool::test_text_to_speech PASSED [ 95%]
tests/integration_tests/apihub/test_translation.py::TestRemoteTool::test_tool FAILED [100%]

=================================== FAILURES ===================================
_________________________ TestRemoteTool.test_formula __________________________

self = <tests.integration_tests.apihub.test_ocr.TestRemoteTool testMethod=test_formula>

    @pytest.mark.asyncio
    async def test_formula(self):
>       toolkit = RemoteToolkit.from_aistudio("formula", file_manager=self.file_manager)

tests/integration_tests/apihub/test_ocr.py:69: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
    return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
    toolkit = RemoteToolkit.from_openapi_dict(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'erniebot_agent.tools.remote_toolkit.RemoteToolkit'>
openapi_dict = {'errorCode': 500, 'errorMsg': '工具不存在', 'logId': '8b8c797e20df8dc01527aff20169f34c', 'servers': [{'url': 'https://tool-formula.aistudio-hub.baidu.com'}]}
access_token = '4ce50e3378f418d271c480c8ddfa818537071dbe'
file_manager = <erniebot_agent.file.file_manager.FileManager object at 0x106cc2470>

    @classmethod
    def from_openapi_dict(
        cls,
        openapi_dict: Dict[str, Any],
        access_token: Optional[str] = None,
        file_manager: Optional[FileManager] = None,
    ) -> RemoteToolkit:
>       info = EndpointInfo(**openapi_dict["info"])
E       KeyError: 'info'

src/erniebot_agent/tools/remote_toolkit.py:175: KeyError
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-71' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.467 seconds
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-71' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.228 seconds
_____________________ TestRemoteTool.test_shopping_receipt _____________________

self = <tests.integration_tests.apihub.test_ocr.TestRemoteTool testMethod=test_shopping_receipt>

    @pytest.mark.asyncio
    async def test_shopping_receipt(self):
>       toolkit = RemoteToolkit.from_aistudio("shopping-receipt", file_manager=self.file_manager)

tests/integration_tests/apihub/test_ocr.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
    return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
    toolkit = RemoteToolkit.from_openapi_dict(
src/erniebot_agent/tools/remote_toolkit.py:185: in from_openapi_dict
    parameter_view = ToolParameterView.from_openapi_dict(schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'erniebot_agent.tools.schema.ToolParameterView'>
schema = {'properties': {'log_id': {'type': 'integer'}, 'pdf_file_size': {'type': 'string'}, 'words_result': {'properties': {'c...d', 'pdf_file_size', 'words_result'], 'type': 'object', 'x-apifox-orders': ['log_id', 'pdf_file_size', 'words_result']}

    @classmethod
    def from_openapi_dict(cls, schema: dict) -> Type[ToolParameterView]:
        """parse openapi component schemas to ParameterView
        Args:
            response_or_returns (dict): the content of status code
    
        Returns:
            _type_: _description_
        """
    
        # TODO(wj-Mcat): to load Optional field
        fields = {}
        for field_name, field_dict in schema.get("properties", {}).items():
            # skip loading invalid field to improve compatibility
            if "type" not in field_dict:
                raise ToolError(f"`type` field not found in `{field_name}` property", stage="Loading")
    
            if "description" not in field_dict:
>               raise ToolError(f"`description` field not found in `{field_name}` property", stage="Loading")
E               erniebot_agent.utils.exceptions.ToolError: An error occured in stage <Loading>. The error message is `description` field not found in `log_id` property

src/erniebot_agent/tools/schema.py:250: ToolError
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-95' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 1.080 seconds
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-95' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.310 seconds
___________________________ TestRemoteTool.test_tool ___________________________

self = <tests.integration_tests.apihub.test_pic_translate.TestRemoteTool testMethod=test_tool>

    @pytest.mark.asyncio
    async def test_tool(self):
        toolkit = RemoteToolkit.from_aistudio("pic-translate", file_manager=self.file_manager)
    
        file = await self.file_manager.create_file_from_path(self.download_fixture_file("shouxiezi.png"))
        agent = self.get_agent(toolkit)
    
>       result = await agent.run("这张照片里面讲了啥？", files=[file])

tests/integration_tests/apihub/test_pic_translate.py:18: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/agents/agent.py:129: in run
    raise e
src/erniebot_agent/agents/agent.py:126: in run
    agent_resp = await self._run(prompt, files)
src/erniebot_agent/agents/function_agent.py:149: in _run
    curr_step, new_messages = await self._step(chat_history)
src/erniebot_agent/agents/function_agent.py:190: in _step
    tool_resp = await self.run_tool(tool_name=tool_name, tool_args=tool_args)
src/erniebot_agent/agents/agent.py:176: in run_tool
    raise e
src/erniebot_agent/agents/agent.py:173: in run_tool
    tool_resp = await self._run_tool(tool, tool_args)
src/erniebot_agent/agents/agent.py:251: in _run_tool
    tool_ret = await tool(**parsed_tool_args)
src/erniebot_agent/tools/remote_tool.py:139: in __call__
    tool_arguments = await self.__pre_process__(tool_arguments)
src/erniebot_agent/tools/remote_tool.py:117: in __pre_process__
    tool_arguments = self.tool_view.parameters(**tool_arguments).model_dump(mode="json")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OpenAPIParameterView(image=b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x03\x06\x00\x00\x03\xf8\x08\x02\x00\x00\x00\x...\xd8W\xe2\x8b\xe6k7c\xaeN+\x03.\xd7\xfe?eP\xf75\xb6\xa4\x95L\x00\x00\x00\x00IEND\xaeB`\x82', from='auto', to='zh', v=3)

    def model_dump(
        self,
        *,
        mode: Literal['json', 'python'] | str = 'python',
        include: IncEx = None,
        exclude: IncEx = None,
        by_alias: bool = False,
        exclude_unset: bool = False,
        exclude_defaults: bool = False,
        exclude_none: bool = False,
        round_trip: bool = False,
        warnings: bool = True,
    ) -> dict[str, Any]:
        """Usage docs: https://docs.pydantic.dev/2.5/concepts/serialization/#modelmodel_dump
    
        Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
    
        Args:
            mode: The mode in which `to_python` should run.
                If mode is 'json', the dictionary will only contain JSON serializable types.
                If mode is 'python', the dictionary may contain any Python objects.
            include: A list of fields to include in the output.
            exclude: A list of fields to exclude from the output.
            by_alias: Whether to use the field's alias in the dictionary key if defined.
            exclude_unset: Whether to exclude fields that have not been explicitly set.
            exclude_defaults: Whether to exclude fields that are set to their default value from the output.
            exclude_none: Whether to exclude fields that have a value of `None` from the output.
            round_trip: Whether to enable serialization and deserialization round-trip support.
            warnings: Whether to log warnings when invalid fields are encountered.
    
        Returns:
            A dictionary representation of the model.
        """
>       return self.__pydantic_serializer__.to_python(
            self,
            mode=mode,
            by_alias=by_alias,
            include=include,
            exclude=exclude,
            exclude_unset=exclude_unset,
            exclude_defaults=exclude_defaults,
            exclude_none=exclude_none,
            round_trip=round_trip,
            warnings=warnings,
        )
E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid utf-8

../../../../miniconda3/envs/eb-agent/lib/python3.10/site-packages/pydantic/main.py:308: UnicodeDecodeError
----------------------------- Captured stderr call -----------------------------
�[92mINFO - [Run][Start] FunctionAgent is about to start running with input:
�[94m这张照片里面讲了啥？�[92m�[0m
�[92mINFO - [LLM][Start] ERNIEBot is about to start running with input:
 role: �[94muser�[92m 
 content: �[94m这张照片里面讲了啥？
<file>file-local-da8582b4-b3a6-11ee-803b-eac7af941eda</file>�[92m �[0m
�[92mINFO - [LLM][End] ERNIEBot finished running with output:
 role: �[93massistant�[92m 
 function_call: �[93m
{
  "name": "pic-translate/v1.0/pic_translate",
  "thoughts": "用户想要知道图片中的文字内容，我需要调用OCR工具来识别图片中的文字，然后根据需要翻译成中文。任务拆解：[sub-task1: 使用OCR工具识别图片中的文字，sub-task2: 根据需要将文字翻译成中文]。接下来，我需要调用[pic-translate/v1.0/pic_translate]来翻译图片中的文字。",
  "arguments": "{\"image\":\"file-local-da8582b4-b3a6-11ee-803b-eac7af941eda\",\"to\":\"zh\"}"
}�[92m �[0m
�[92mINFO - [Tool][Start] �[95mRemoteTool�[92m is about to start running with input:
�[95m{
  "image": "file-local-da8582b4-b3a6-11ee-803b-eac7af941eda",
  "to": "zh"
}�[92m�[0m
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-98' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:101> wait_for=<Future pending cb=[shield.<locals>._outer_done_callback() at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/tasks.py:864, Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 1.090 seconds
___________________________ TestRemoteTool.test_tool ___________________________

self = <tests.integration_tests.apihub.test_translation.TestRemoteTool testMethod=test_tool>

    @pytest.mark.asyncio
    async def test_tool(self):
>       toolkit = RemoteToolkit.from_aistudio("translation", file_manager=self.file_manager)

tests/integration_tests/apihub/test_translation.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
    return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
    toolkit = RemoteToolkit.from_openapi_dict(
src/erniebot_agent/tools/remote_toolkit.py:185: in from_openapi_dict
    parameter_view = ToolParameterView.from_openapi_dict(schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'erniebot_agent.tools.schema.ToolParameterView'>
schema = {'properties': {'log_id': {'description': 'logId', 'type': 'integer'}, 'result': {'properties': {'from': {'description... '翻译结果数组列表', 'items': {'properties': {...}, 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'}

    @classmethod
    def from_openapi_dict(cls, schema: dict) -> Type[ToolParameterView]:
        """parse openapi component schemas to ParameterView
        Args:
            response_or_returns (dict): the content of status code
    
        Returns:
            _type_: _description_
        """
    
        # TODO(wj-Mcat): to load Optional field
        fields = {}
        for field_name, field_dict in schema.get("properties", {}).items():
            # skip loading invalid field to improve compatibility
            if "type" not in field_dict:
                raise ToolError(f"`type` field not found in `{field_name}` property", stage="Loading")
    
            if "description" not in field_dict:
>               raise ToolError(f"`description` field not found in `{field_name}` property", stage="Loading")
E               erniebot_agent.utils.exceptions.ToolError: An error occured in stage <Loading>. The error message is `description` field not found in `result` property

src/erniebot_agent/tools/schema.py:250: ToolError
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-152' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.271 seconds
=========================== short test summary info ============================
FAILED tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_formula
FAILED tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_shopping_receipt
FAILED tests/integration_tests/apihub/test_pic_translate.py::TestRemoteTool::test_tool
FAILED tests/integration_tests/apihub/test_translation.py::TestRemoteTool::test_tool
=================== 4 failed, 20 passed in 237.27s (0:03:57) ===================

[wj-Mcat]

以上报错的几个 tool 是由于 yaml 内容错误导致的。

也就是现在的 PR 能够兼容之前tool 的文件解析格式。

[Bobholamovic]

File部分LGTM